How to Set Up a Self-Hosted Eval Gate for AI Output

A self-hosted eval gate is a test process under your control that returns pass or fail before AI-generated output can move to the next stage. It does not require the generator to be self-hosted. The gate can run on a workstation or private CI runner, keep test data inside your environment, and call a local judge model only when deterministic checks cannot decide the result.

The cited product and tool specifications are as of 2026-07-20 and may change.

The core design is simple: hard checks first, rubric judgment second, human review for uncertain or high-risk cases. An LLM judge should not be the only gate.

1. Define the contract

Start with a small set of representative cases in JSON Lines format:

{"id":"support-001","input":"How do I reset my password?","must_include":["account settings"],"must_not_include":["send me your password"],"max_chars":800,"rubric":"Give safe, actionable steps without requesting credentials."}
{"id":"extract-001","input":"Extract the invoice fields","schema":"invoice.schema.json","rubric":"Preserve values exactly and do not invent missing fields."}

Each case should contain an input, machine-checkable constraints, and a short rubric for qualities that cannot be reduced to a string or schema. Keep expected behavior in version control. Add a regression case whenever a real failure escapes.

2. Separate generation from evaluation

Store the candidate output rather than asking the judge to regenerate it:

evals/
  cases.jsonl
  rubrics/
  schemas/
  run_eval.py
  judge.py
artifacts/
  candidates.jsonl
  results.jsonl

This separation makes a failed run reproducible. Record the generator model, prompt version, parameters, commit SHA, timestamp, and output. Do not log secrets or unnecessary personal data.

3. Run deterministic gates first

Use ordinary code for facts that code can decide:

• valid JSON and schema conformance;
• required and forbidden strings;
• file type, length, and language checks;
• unit tests, linters, type checks, and security scanners for code;
• citation presence and URL allowlists for research output;
• exact arithmetic or database lookups for known answers.

A minimal Python runner can return a nonzero exit status when any hard rule fails:

def hard_checks(case, output):
    failures = []
    if len(output) > case.get("max_chars", 10**9):
        failures.append("too_long")
    for text in case.get("must_include", []):
        if text.lower() not in output.lower():
            failures.append(f"missing:{text}")
    for text in case.get("must_not_include", []):
        if text.lower() in output.lower():
            failures.append(f"forbidden:{text}")
    return failures

Wrap the runner in pytest or call it directly from CI. Pytest defines exit code 0 as all collected tests passing and exit code 1 as test failures, which makes it suitable for a build gate (pytest exit codes).

4. Add a rubric-based local judge

Use a judge only for dimensions such as relevance, clarity, groundedness, or policy adherence. Run an open-weight instruction model behind a local endpoint, then require structured output:

{
  "pass": true,
  "scores": {"correctness": 4, "relevance": 4, "safety": 5},
  "evidence": ["The answer directs the user to account settings."],
  "failures": []
}

The judge prompt should include only the task, candidate, reference evidence, and explicit scoring anchors. For example, define correctness 5 as “all material claims supported by the supplied reference” and correctness 1 as “a central claim contradicts the reference.” Avoid a vague request to “rate quality.”

Require the judge to cite spans from the candidate or reference for each failure. Validate the returned JSON before using it. If parsing fails, the gate should fail closed or route to review rather than silently pass.

5. Calibrate the judge before trusting it

Create a labeled calibration set with clear passes, clear failures, and boundary cases. Compare judge decisions with human labels and report false-pass and false-fail rates separately. For a release gate, false passes usually carry more risk.

Run pairwise comparisons in both orders when comparing two candidates. Research has documented position bias in LLM judges (Shi et al., 2024) and superficial-quality or verbosity bias (Zhou et al., 2024). Mitigations include explicit criteria, order swapping, evidence requirements, and human adjudication for disagreements. They reduce risk; they do not make the judge authoritative.

Do not let the same judge invent the rubric, label the calibration set, and approve its own performance. Human owners should define the acceptance contract.

6. Set a release policy

A practical policy might be:

hard_checks: all_pass
judge:
  correctness_min: 4
  safety_min: 5
  overall_min: 4
uncertainty:
  missing_evidence: human_review
  judge_disagreement: human_review
regression:
  allowed_false_pass_rate: 0.01

Do not average away a critical failure. A perfect style score cannot compensate for invalid JSON, a leaked secret, or an unsupported medical claim. Keep “must pass” dimensions separate from aggregate quality scores.

7. Wire it into CI

The CI job should:

1. generate or load candidate artifacts;
2. run deterministic checks;
3. run the local judge only on remaining cases;
4. compare aggregate and per-slice results with the approved baseline;
5. write a machine-readable report;
6. exit nonzero on policy failure;
7. retain enough metadata to reproduce the decision.

Pin the judge model, runtime, prompt, and rubric versions. Report results by task type and risk level; a single average can hide a collapse in one important slice. Keep a manual override rare, named, time-limited, and recorded with a reason.

8. Maintain the gate

Review failures monthly or after material model changes. Add escaped defects to the test set, but hold back a private set so the generator is not optimized only for visible cases. Recalibrate when changing the judge model or prompt. Monitor judge parse failures, disagreement rates, and score drift.

9. Common failure modes

A few patterns break eval gates in practice. The first is letting the judge score dimensions a deterministic check could settle: schema validity, link resolution, or forbidden strings belong in code, not in a model prompt. The second is averaging a safety failure into an aggregate score, which lets a high style rating mask an unsupported claim. The third is optimizing the generator against the visible test set until the gate passes but real quality stalls; a held-back private set is the guard against that. The fourth is silent drift — the judge model or prompt changes and the old thresholds no longer mean what they did, yet recalibration is skipped. The last is treating a green gate as proof of correctness rather than proof that the known checks passed. Name what the gate does not cover, and route uncertainty to a person rather than to a default pass.

The finished gate is not the judge prompt. It is the complete chain of deterministic verification, calibrated judgment, explicit thresholds, reproducible artifacts, and human handling for uncertainty. That structure makes AI output testable without turning another model into an unquestioned authority.