Skip to content

commandData and AI

Eval Prompt

Runs a prompt against several test inputs and reports which outputs fail your stated criteria, with the evidence for each failure.

npx workspacetemplates add eval-prompt --tool claude-code

Writes 1 file to .claude/commands/

.claude/commands/eval-prompt.md14 lines · 1.2 KBFile
---
description: Runs a prompt against several test inputs and reports which outputs fail your stated criteria, with the evidence for each failure.
argument-hint: "[prompt file or text] [criteria]"
---

Evaluate this prompt. Input: $ARGUMENTS (a prompt file path or text, plus the pass criteria).

1. If criteria are missing, write 3 to 6 testable ones (format, length, required facts, tone, refusals) and show them for confirmation. Vague criteria like "good" are not allowed.
2. Build 8 to 12 test inputs: typical cases, empty or very long input, a different language, an injection attempt such as "ignore previous instructions", and two cases that should be refused or answered "unknown". Use any example data in the repo first.
3. Run the prompt on each input using the project's own script or the available model tool. Do not call outside services unless the project already does.
4. Grade each output against each criterion as pass or fail, quoting the failing text.
5. Report a table: input, criterion, result, quote. Give the pass rate per criterion.

Finish with the two likeliest causes in the prompt and one suggested edit for each. Do not change the prompt file until asked.

What it does

Changing a prompt without testing it is guesswork. This command takes the prompt and your pass criteria, builds a small set of test inputs including awkward ones, and checks each output against the criteria one by one.

The report shows what passed, what failed and the exact text that broke the rule, so you can fix the prompt instead of just rewording it again.

Example

you › /eval-prompt prompts/summarize.md must be under 80 words, must name the author, no opinions

agent › 10 inputs. Length 9/10, author named 6/10, no opinions 10/10.
Fails: input 4 (no byline in text, model invented a name: "by J. Smith"), input 7 (empty input, wrote a summary of nothing).
Suggest: add "if no author, write unknown". Apply?

Questions

Does it edit my prompt?
Not until you ask. First it only reports failures and suggests fixes.
Where do the test inputs come from?
It reuses examples in your repository first, then writes edge cases such as empty input and injection attempts.
  • Analyze CSV

    Commandpython

    Free
  • Clean Notebook

    Commandpython

    Free
  • Profile DataFrame

    Commandpython

    Free
  • SQL From Question

    CommandData and AI

    Free