Skip to content

skillData and AIpythonpandas

Pandas Data Cleaning

Cleans messy DataFrames with typed columns, explicit null handling and assertions. Use it when raw CSV or Excel data needs to become trustworthy.

npx workspacetemplates add pandas-data-cleaning --tool claude-code

Writes 1 file to .claude/skills/pandas-data-cleaning/

.claude/skills/pandas-data-cleaning/SKILL.md28 lines · 1.6 KBFile
---
name: pandas-data-cleaning
description: Cleans messy DataFrames with typed columns, explicit null handling and assertions. Use it when raw CSV or Excel data needs to become trustworthy.
---

# Pandas data cleaning

Every change to the data must be explicit, counted and checked.

## Steps

1. Load with intent: pass `dtype=` for IDs and codes (keep them as strings so leading zeros survive), `parse_dates=` for known dates, and `na_values=` for the file's own null markers such as "N/A" or "-".
2. Look before you fix: `df.info()`, `df.isna().sum()`, `df.nunique()`, and `df.describe(include="all")`. Print the number of rows at the start.
3. Normalize text columns: `.str.strip()`, consistent case, and one canonical spelling map for categories.
4. Convert types explicitly with `pd.to_numeric(errors="coerce")` and `pd.to_datetime(errors="coerce")`, then count what became null and inspect those rows before deciding.
5. Decide each null on purpose: drop, fill with a stated default, or keep as `pd.NA` with a nullable dtype (`Int64`, `string`, `boolean`).
6. Deduplicate on a named key with `drop_duplicates(subset=[...], keep=...)` and say which row wins and why.
7. Finish with assertions: unique key, no nulls in required columns, values inside allowed ranges, expected dtypes.

## Rules

- Never use `inplace=True` or chained assignment. Return a new frame from a pure function.
- Never drop rows silently. Log how many were removed per step.
- Do not guess a date format; use an explicit `format=`.

## Report

Rows in and out, per step counts, columns whose type changed, and any rows set aside for a human to review.

What it does

Quiet pandas defaults are where bad data hides: strings that look like numbers, NaN that turns an integer column into floats, duplicates nobody counted. This skill writes cleaning code that states every decision and fails loudly when the data breaks an assumption.

The result is a short, readable function per dataset, a log of rows dropped or changed, and assertions you can keep running on the next file.

Example

you › Clean customers_raw.csv so signup_date is a date and customer_id is unique.

agent › Wrote clean_customers() in etl/customers.py. 12,480 rows in, 12,211 out: 214 exact duplicates and 55 rows with unparseable signup_date moved to rejects.csv. customer_id kept as string, phone numbers stripped to digits, plan mapped from 9 spellings to 3. Added 5 assertions that run at the end of the function.

Questions

Why keep IDs as strings?
Leading zeros and long numbers are lost the moment pandas reads them as integers or floats. Strings keep them exact and safe to join on.
Does it modify my source file?
No. It reads the raw file and writes cleaned output and a rejects file next to it, leaving the original untouched.
  • Data Pipeline Design

    SkillData and AI

    Free
  • dbt Modeling

    SkillData and AI

    Free