Skip to content

skillData and AIpython

Model Evaluation

Chooses metrics, splits and baselines for a model, then reports results with uncertainty. Use it before claiming a model is good or better than another.

npx workspacetemplates add model-evaluation --tool claude-code

Writes 1 file to .claude/skills/model-evaluation/

.claude/skills/model-evaluation/SKILL.md28 lines · 1.8 KBFile
---
name: model-evaluation
description: Chooses metrics, splits and baselines for a model, then reports results with uncertainty. Use it before claiming a model is good or better than another.
---

# Model evaluation

Report what a skeptical reviewer would accept, not the best-looking number.

## Steps

1. State the decision the model supports and the cost of each error type. Choose the metric from that: precision and recall or PR AUC for rare positives, MAE or RMSE for regression (pick by outlier cost), calibration for probabilities used as such. Accuracy alone is wrong for imbalanced classes.
2. Split to match deployment. Use time-based splits for anything that predicts the future, group splits when rows share an entity (user, patient), and stratified splits otherwise. Fit all preprocessing on the training part only.
3. Keep three sets: train, validation for tuning, and a test set touched once at the end.
4. Build baselines first: majority class or mean, then a simple model such as logistic regression or last value. A complex model must beat these.
5. Run cross validation or several seeds. Report the mean and the spread, and a bootstrap confidence interval for the headline metric.
6. Slice results: by segment, by time period and by input difficulty. Look at the worst slice and at a sample of the worst errors.
7. For classifiers, check the threshold choice and a calibration curve.
8. Check leakage: features derived from the target, duplicates across splits, and suspiciously perfect scores.

## Rules

- Never tune against the test set.
- Never report a gain smaller than the run to run noise as an improvement.

## Report

A table of model against baseline with intervals, the slice results, the three most telling error examples, and a plain statement of what the numbers do not show.

What it does

A model score means little without the right metric, a split that matches real use, and a baseline to beat. This skill sets those up before it computes a single number, and then reports the result with its uncertainty and its weak spots.

It also looks for the usual traps: leakage between train and test, tuning on the test set, and an average that hides a failing segment.

Example

you › Evaluate the churn model against last quarter's data.

agent › Time split: trained to June, tested July to September. Baseline (predict no churn) has 0.00 recall. Logistic regression PR AUC 0.31; gradient boosting 0.34 (95% CI 0.32 to 0.36), a real but small gain. Recall at 10% review rate: 41%. Weak slice: accounts under 30 days old, PR AUC 0.12. No leakage found; tenure feature checked.

Questions

Why does it insist on a baseline?
Without one, a score has no meaning. A baseline shows how much of the result comes from the model and how much was easy to begin with.
Will it retrain my model?
It trains small baselines for comparison. Your own model is loaded and scored, not changed.
  • Data Pipeline Design

    SkillData and AI

    Free
  • dbt Modeling

    SkillData and AI

    Free