Skip to content

Evals 0.9.0 judge reliability and live resilience (Tower 0.7.1)

Agentic Evals 0.9.0 adds optional fields to native report schema 1.0. They record judge cost, retries, unavailable judgments and sampling agreement, and live-target timeouts and errors per case. Control Tower 0.7.1 bundles the exact Evals 0.9.0 report and gate schemas and validates complete reports and supplied gates against them before projecting evidence, as before.

Evidence details for an agentic-evals evaluation report now also carry:

Projected field Source Meaning
summary.judge.judge_scores, unavailable_scores, unavailable_cases summary.judge judgments made, and those the judge could not deliver
summary.judge.total_cost_usd, min_agreement summary.judge judge spend (null unless every judge score was priced) and the lowest sampling agreement
summary.live.retried_cases, timed_out_cases, error_cases and their case IDs summary.live target outages, kept apart from wrong answers

Both are null in reports written before Evals 0.9.0. summary.live is also null for offline (non-live) evaluations, and summary.judge is null when a report has no judge scores; an offline evaluation with LLM judges does carry summary.judge, and Tower projects it. Tower shows a missing summary as missing, not as healthy.

The producer's gate stays authoritative. Evals 0.9.0 fails its default gate when any judgment is unavailable, and Tower carries gate.passed and reasons unchanged without recomputing them. An unavailable judge, a timed-out case or a passing gate is evaluation evidence. None of them is deployment health, recovery authorization or proof of containment.

The base package still imports no sibling. The optional agentic-evals extra keeps >=0.7.2,<1; only the contract-test dev pin moves to 0.9.0. See the Evals guide.