Skill
Data Pipeline Design
Designs idempotent, testable batch pipelines with safe backfills and data quality checks. Use it when planning or reviewing a scheduled data job.
1Design for the rerun. Assume every job will fail midway and be started again.
2Steps
31. Define the contract: source, destination, schedule, the grain of the output, and who consumes it. Write the output schema with types and a unique key.
42. Partition by processing window (usually a date). Each run takes the window as a parameter and never reads "now" or "latest" implicitly.
53. Make writes idempotent: overwrite the partition, or `MERGE`/upsert on the key. Write to a staging location, validate, then swap or publish atomically. Never append blindly.
Claude CodeCodex
Data Pipeline Design
SkillData and AI
Free