An LLM evaluation dataset should help a team make a specific decision, such as whether a prompt change is safe to release. It should not be a scrapbook of awkward outputs or a generic test of model intelligence. The most useful starting material is the system’s own failures: incorrect answers, unsupported claims, malformed tool calls, unsafe actions, needless refusals, missed constraints, and costly retries.
The practical workflow is to capture incidents safely, reduce each one to a reproducible case, define success, and protect part of the dataset from repeated tuning. It will not predict every failure, but it can make known failure modes measurable and harder to reintroduce.
The original evaluation practices and data-handling guidance cited below were checked against official sources on 2026-07-23; the additional Anthropic and Google Cloud guidance was checked on 2026-07-24. No universal dataset size, split ratio, or pass threshold is assumed.
Define the decision the eval supports
Start with the decision, not the examples. “Evaluate our assistant” is too broad to determine what belongs in the dataset or how results should block a release. A better statement is:
Decide whether prompt, model, retrieval, or tool-schema version X can replace the current production configuration without increasing critical failure rates.
Write down the system boundary. If a case fails, the team must know whether the dataset tests an end-to-end path or one component. A retrieval-only dataset, for example, should measure ranked evidence rather than generated prose; the RAG retrieval evaluation guide covers that narrower contract.
For each proposed evaluation, record:
- the release or operating decision it informs;
- the system version and components under test;
- the input unit, such as one request, conversation, document, or agent task;
- the behaviors that count as success, acceptable variation, and failure;
- critical slices, such as a language, customer workflow, permission level, or tool;
- failures that must block release regardless of the average score;
- the owner who resolves ambiguous labels.
This prevents metric shopping. A higher style score cannot compensate for unauthorized tool execution, and prose elegance should not dominate an extraction test.
NIST’s AI Risk Management Framework says test sets, metrics, and evaluation tools should be documented, and that performance should be assessed under conditions similar to deployment. That supports a decision-specific suite rather than a context-free leaderboard (official NIST AI RMF Core).
Collect representative failures safely
Useful cases can come from production logs, support tickets, incident reviews, user corrections, human review, rejected tool approvals, and staging tests. Use several channels: tickets overrepresent visible problems, while automated logs can miss subtly wrong answers.
Create a small incident envelope before moving anything into the evaluation store:
incident_id: inc-2026-0042
observed_at: 2026-07-18
system_version: prompt-31_retriever-12
failure_family: unsupported_claim
severity: high
source_channel: reviewed_production_trace
input_ref: redacted-copy-0042
observed_behavior: "Answer asserted a policy exception without evidence."
expected_behavior: "Cite supporting policy text or abstain."Preserve only facts needed to reproduce the failure: system version, inputs, context IDs, tool responses, permissions, locale, and relevant configuration. A full trace may contain names, account identifiers, secrets, private documents, or unrelated turns.
Confirm that the organization may reuse the data, remove irrelevant fields and credentials, replace identifying values with consistent placeholders, restrict raw-incident access, and retain a deletion path. Keep sanitized cases separate from source records.
The European Commission’s official GDPR guidance states that personal data should be adequate, relevant, and limited to what is necessary for the purpose, and prefers anonymous data where possible (official data-minimisation guidance). This article is an engineering workflow, not legal advice; applicable rules and contractual duties still require local review.
Do not let one customer or incident flood the suite with duplicates. Deduplicate by root cause, preserve meaningful variants, and sample ordinary successful traffic. A failure-only dataset can reward needless refusals.
Turn incidents into test cases
An incident is historical evidence; a test case is a controlled experiment. Retain the triggering condition, and make the case run without a deleted ticket, changing page, or mutable production record.
A portable case schema might include:
id: unsupported-policy-017
task: answer_with_evidence
input:
user_message: "Can contractors keep access after the review expires?"
context_fixture:
- "Contractor access ends when the scheduled review expires."
expected:
required_behavior: "State that access ends at expiry."
citation_required: true
forbidden_behavior:
- "Invent an exception."
metadata:
failure_family: unsupported_claim
severity: high
source: sanitized_production_incident
source_incident_ids: ["inc-2026-0042"]Keep inputs, expected behavior, and metadata separate so the evaluator does not expose the answer or rubric to the system. Store stable fixtures rather than live URLs. For an agent, use simulated tool results so a rerun cannot alter a real account.
Reduce one incident into the smallest case that still fails on the old configuration. Then add a limited number of purposeful variants:
- a paraphrase that preserves the same intent;
- a normal case that should succeed without abstaining;
- a boundary case that changes the correct outcome;
- a permission variant that must not reveal protected data;
- a malformed or missing field when input quality caused the failure.
Variants test whether a fix generalizes. Review synthetic cases because they often inherit the generator’s vocabulary and become cleaner than real requests.
Tag both symptom and probable root cause. “Wrong answer” is a symptom; retrieval_miss, instruction_conflict, and tool_schema_mismatch are actionable families. Keep the cause revisable.
For deterministic constraints such as valid JSON, allowed tool names, or required citations, turn the case into a mechanical assertion. The machine-gate guide explains how to keep such checks separate from subjective review.
Write expected results and rubrics
Use the least subjective grader that correctly represents the requirement. Exact match is suitable for a normalized identifier, not for an explanatory answer with many valid phrasings. A schema validator can grade structure. Code can check tool name, argument types, citations, refusal markers, or forbidden data. Human or model-based grading is more appropriate for semantic correctness, completeness, and tone.
Every rubric should define observable evidence. Replace “good answer” with criteria such as:
correctness: all material claims agree with the supplied reference;support: each factual claim is supported by the provided context;completeness: the answer includes the two required actions;boundary: the response does not expose restricted fields or call a disallowed tool;calibration: the system abstains when the fixture lacks sufficient evidence.
A simple ordinal scale can be useful, but define every level:
2 = pass: correct outcome and no material violation
1 = partial: useful but missing a non-critical required element
0 = fail: wrong outcome, unsupported material claim, or unusable response
CRITICAL = security, privacy, or irreversible-action violationDo not average CRITICAL into the ordinary score. Make it an independent release condition and retain grader explanations for audit.
Calibrate on a varied sample. Have qualified reviewers grade the same cases independently and resolve disagreements before scaling. Compare an LLM grader with human decisions on difficult cases, and record its version and prompt.
OpenAI’s official evaluation flywheel describes manual failure analysis, a test dataset, graders, and targeted improvement as a repeating process (official OpenAI Cookbook). Its Evals API also treats an evaluation as testing criteria plus a defined data-source schema (official Evals API reference). Those components should be versioned together.
Anthropic’s official guidance recommends task-specific evaluations that reflect the real task distribution and edge cases. It also recommends testing an LLM grader’s reliability before scaling it (official Anthropic evaluation guidance, checked 2026-07-24). In practice, start with mechanical checks where the requirement is deterministic, and validate subjective graders against reviewed examples.
Choose the grading method by requirement
| Requirement | Preferred grader | Evidence to retain | Main limitation |
|---|---|---|---|
| Exact identifier or label | Normalized exact match | Expected and actual value | Rejects valid variants unless normalization is defined |
| JSON or tool-call contract | Schema and rule checks | Validation errors, tool name, and arguments | Does not judge whether a structurally valid action is useful |
| Evidence-grounded answer | Claim or citation checks, then reviewed semantic grading | Source fixture, cited spans, and grader explanation | Automated semantic graders can disagree with expert review |
| Completeness or tone | Human-calibrated rubric or validated LLM grader | Per-criterion score and explanation | More subjective and sensitive to rubric wording |
| Privacy, authorization, or irreversible action | Deterministic policy checks plus qualified review | Violated rule, trace, and reviewer decision | A single average score can hide a critical failure |
Use more than one grader when one requirement has both mechanical and semantic parts. For example, a tool call can pass its JSON schema and still select the wrong tool. Keep each result separate so the report shows which contract failed.
Google Cloud’s official judge-model documentation uses human ratings as ground truth when assessing model-based metrics, including pointwise and pairwise judgments (official Google Cloud judge-model evaluation guide, checked 2026-07-24). This does not make human labels infallible; it makes disagreements visible and gives the team a basis for deciding where automated grading is reliable enough.
Split calibration and holdout sets
Use a calibration set while designing prompts, retrieval, tools, and rubrics. Use a holdout set only after the candidate configuration and decision rules are fixed. Repeatedly inspecting holdout failures turns the holdout into another tuning set, even if no one calls it that.
Split related examples as a group. An incident, its paraphrases, and boundary variants belong on the same side. Group by workflow, document lineage, or conversation when those fields create near-duplicates.
Consider a time-based holdout when production conditions drift. Older failures can support calibration while newer, untouched incidents test whether the fix generalizes. For rare critical risks, deliberately place representative cases in both sets, but use distinct incidents rather than copied variants.
There is no universal split percentage. The holdout must be large and diverse enough to support the decision, while the calibration set must expose enough failure families to guide improvement. Record the rationale instead of copying a conventional ratio.
Google’s official machine-learning guidance recommends separate training, validation, and test data, warns that validation and test sets can wear out through repeated use, and calls for removal of duplicates across splits (official dataset-splitting guidance). For an LLM application, calibration plays a role similar to validation even when the team is changing prompts and orchestration rather than model weights.
Restrict holdout access where practical. When a holdout case becomes part of debugging, mark it exposed, move it to calibration in the next version, and replace its coverage.
Version data and prevent leakage
Treat the dataset like source code with stricter data controls. Give every case a stable ID, track changes, and publish a manifest containing:
- dataset version and creation date;
- included case IDs and content hashes;
- schema and rubric versions;
- sanitization status and access classification;
- source lineage without raw private records;
- calibration and holdout assignment;
- known limitations and missing slices;
- compatible evaluator and fixture versions.
Leakage can occur through more paths than fine-tuning. A holdout answer may appear in prompt examples, retrieval documents, test fixtures used during development, issue descriptions, shared debugging chats, model-grader prompts, or generated synthetic cases. Search those paths before claiming an independent result.
Separate evaluation data from improvement data. If a case enters few-shot examples, fine-tuning data, or a retrieval corpus, exclude it and close variants from the holdout. Hashes detect duplicates; normalized matching and review help find paraphrases.
Log each run with the application configuration, model identifier, prompt and tool-schema versions, dataset version, evaluator version, randomization settings where available, and raw outputs. This is the evidence needed to explain a score change.
Private data need not be published for reproducibility. Controlled storage, manifests, access logs, and immutable internal artifacts can provide it. For a local gate, see the self-hosted evaluation gate.
Maintain the dataset over time
An evaluation dataset decays as products, policies, and attacks change. Assign an owner and make case intake part of incident closure. The goal is durable coverage, not perpetual growth.
Use a repeatable maintenance loop:
- triage new incidents by severity, frequency, and novelty;
- check whether an existing case already covers the root cause;
- sanitize and reproduce genuinely new failures;
- add them to calibration first and create fresh holdout coverage where needed;
- review rubric disagreements and evaluator drift;
- retire obsolete cases only with a recorded reason;
- publish a new dataset manifest and rerun the current baseline.
Track coverage by failure family and production slice, not only case count. Near-identical formatting checks do not compensate for missing permission or tool-execution cases. Watch for new failures that map to no existing case.
Refresh the suite after meaningful changes: a new tool, retrieval corpus, policy, language, model family, user workflow, or safety boundary. Also review cases when they pass too easily for a long period. Some represent solved regressions worth retaining; others depend on obsolete prompts or fixtures and should be archived.
Keep the release decision, a slice-level report, and a dataset-defect queue from every run. Separate system failures from mislabeled cases, broken fixtures, and unstable graders.
A maintained dataset creates a useful feedback loop: production failures become controlled tests, tests guide targeted changes, and release results reveal where the dataset is weak. The dataset is successful when it makes decisions more repeatable and catches consequential regressions—not when it reaches an impressive size.
Frequently asked questions
How many examples does an LLM evaluation dataset need?
There is no universal minimum that makes an evaluation trustworthy. Start with enough cases to cover each consequential failure family and production slice, then add cases where uncertainty or disagreement remains. Report case counts per slice and confidence limits where appropriate instead of treating one total as proof of coverage.
Should production failures go into the holdout set?
Usually, the incident used to develop a fix belongs in calibration because the team has already inspected it. Put a distinct, unexposed incident or related case in the holdout to test generalization. Keep the original incident and all close variants on the same side of the split.
Can an LLM grade another LLM’s output?
Yes, when the rubric is explicit and the judge has been compared with qualified human decisions on representative and difficult cases. Record the judge model, prompt, rubric, and explanations. Use code-based checks for deterministic requirements, and do not let a model-generated average override a privacy, authorization, or irreversible-action failure.
How do I prevent evaluation data leakage?
Restrict holdout access, group near-duplicates before splitting, and search prompt examples, retrieval content, fine-tuning data, issue trackers, and synthetic-generation inputs for exposed cases. When a holdout case is used for debugging, mark it exposed and replace its coverage in the next dataset version.
What should I do when an evaluation case fails intermittently?
First separate model variability from a broken fixture, rate limit, or unstable grader. Preserve raw outputs and run metadata, then repeat the case under a documented sampling policy. If tool calls are the source of instability, the tool-calling error guide provides a narrower diagnostic workflow.
Related guides
- Use machine gates for AI output for schema, allowlist, and other deterministic checks.
- Build a self-hosted evaluation gate when evaluation data must remain in a controlled environment.
- Evaluate the retrieval stage separately with the RAG retrieval evaluation guide.
- Add adversarial cases with the RAG prompt-injection testing guide.
- Review grounding controls in How to Prevent RAG Hallucinations.