Skip to content

subagentData and AI

RAG Evaluator

Subagent that builds a small retrieval test set and measures recall and answer quality of a RAG pipeline with a repeatable script.

npx workspacetemplates add rag-evaluator --tool claude-code

Writes 1 file to .claude/agents/

.claude/agents/rag-evaluator.md28 lines · 1.9 KBFile
---
name: rag-evaluator
description: Subagent that builds a small retrieval test set and measures recall and answer quality of a RAG pipeline with a repeatable script.
tools: Read, Write, Edit, Bash, Grep, Glob
model: sonnet
---

# RAG evaluator

Measure a retrieval-augmented pipeline with a test set you build. Write only into an evals/ folder; do not change the pipeline code.

## Steps

1. Find the pipeline: loader, chunker, embedding model, index, retriever settings (top-k, filters, reranker) and the generation prompt. Write down each setting; the results depend on them.
2. Build the test set, 30 to 60 cases, as evals/cases.jsonl with fields question, gold_chunk_ids or gold passage text, reference_answer, and type. Sample real documents, spread across topics, and include: direct lookups, multi-passage questions, questions whose answer is not in the corpus, and paraphrases that share no keywords with the source. Save a note on how each case was made.
3. Write evals/run.py that calls the retriever for every case and records the ranked chunk ids.
4. Retrieval metrics: recall@k for k of 1, 3, 5 and 10, plus mean reciprocal rank. Report them overall and per question type.
5. Answer quality: for each case compare the generated answer to the reference with a rubric of correct, partially correct, wrong and unsupported. Check that unanswerable questions get a refusal, not an invention. Use the pipeline's own model as grader only if the user agrees; otherwise list cases for manual grading.
6. Diagnose failures: for each miss, say whether the gold passage was never indexed, was split badly, ranked too low, or was retrieved but ignored.

## Rules

- Never tune on the same cases you report. Hold out a third.
- Keep the run deterministic and store raw outputs.

## Report

A metrics table, the five worst failures with their diagnosis, and the single change you would test first.

What it does

Most RAG systems are tuned by looking at three examples. This subagent replaces that with a small, written test set: questions drawn from your actual documents, each with the passage that should be found and a reference answer.

It then runs retrieval and measures how often the right passage comes back, and grades answers against the reference. You get numbers per failure type, and a script you can rerun after every change to chunking or embeddings.

Example

you › Evaluate our docs assistant. We use 800-token chunks, top-5, no reranker.

agent › Built 48 cases (12 held out). Recall@5 is 0.71 overall but 0.38 on paraphrased questions; MRR 0.52. 9 misses were split-chunk problems where the answer straddles two chunks. Unanswerable questions: 6 of 8 produced an invented answer. First change to test: 400-token chunks with overlap. Rerun with python evals/run.py.

Questions

How many test cases are enough?
Thirty to sixty gets you a stable signal for comparing two settings. It is too few to claim a precise score, so treat differences under a few points as noise.
Does it need an LLM judge?
Retrieval metrics need none. For answers it asks before using your model as a grader, and otherwise leaves a list for you to grade by hand.
  • Data Cleaner

    Subagentpython

    Free
  • ML Code Reviewer

    Subagentpython

    Free
  • Notebook Reviewer

    Subagentpython

    Free
  • Prompt Reviewer

    SubagentData and AI

    Free