Skip to content

skillData and AIpythonpandas

Pandas Performance

Speeds up slow pandas code by measuring first, then vectorizing, shrinking dtypes and chunking reads. Use it when a script is slow or runs out of memory.

npx workspacetemplates add pandas-performance --tool claude-code

Writes 1 file to .claude/skills/pandas-performance/

.claude/skills/pandas-performance/SKILL.md27 lines · 1.6 KBFile
---
name: pandas-performance
description: Speeds up slow pandas code by measuring first, then vectorizing, shrinking dtypes and chunking reads. Use it when a script is slow or runs out of memory.
---

# Pandas performance

Measure, change one thing, measure again, and prove the output did not change.

## Steps

1. Build a representative sample and keep the current output as the reference. Time the slow path with `time.perf_counter()` or `%timeit`, and record memory with `df.memory_usage(deep=True).sum()`.
2. Profile to find the hot line (`python -m cProfile -s cumtime script.py`). Do not optimize lines that are not hot.
3. Replace row loops. `iterrows`, `itertuples` and `apply(axis=1)` become column arithmetic, `np.where`, `np.select`, `.map` with a dict, or `merge` for lookups.
4. Shrink dtypes: `category` for low-cardinality strings, `int32` or `float32` where range allows (check with `pd.to_numeric(downcast=...)`), nullable ints only when nulls exist, and `datetime64` instead of strings.
5. Read less: pass `usecols=`, set `dtype=` at load, and read big files with `chunksize=` or switch to Parquet. Aggregate per chunk, then combine.
6. Avoid repeated `concat` in a loop; collect pieces in a list and concat once. Filter before merging, not after.
7. Compare the new result with the reference using `pd.testing.assert_frame_equal`.

## Rules

- No optimization without a measurement showing it helps.
- If the data is beyond one machine, say so and name polars or DuckDB rather than forcing pandas.

## Report

A table of step, time and memory before and after, the one change that mattered most, and the equality check result.

What it does

Most slow pandas code is slow for one of three reasons: a Python loop over rows, columns stored in a wasteful type, or a file read all at once. This skill finds which one applies by profiling, and only then changes the code.

Each change is checked against the original output so the faster version returns the same numbers. You get timings before and after, not a guess.

Example

you › build_features.py takes 40 minutes on the full events file. Make it faster.

agent › Profiling showed 91% of time in apply(axis=1) at line 58. Replaced with np.select and a merge on the plan table. Converted user_id and country to category and read with usecols. Runtime 40 min to 2 min 10 s, memory 11.2 GB to 3.4 GB. assert_frame_equal passes against the saved reference.

Questions

Will it switch me to polars?
Not unless pandas cannot do the job. It stays in pandas, and only names an alternative when the data clearly outgrows it.
How does it make sure results stay the same?
It saves the original output first and compares the new output to it with a frame equality assertion after every change.
  • Data Pipeline Design

    SkillData and AI

    Free
  • dbt Modeling

    SkillData and AI

    Free