Release Eval Scorecard

Source Code: src/gaia/eval/release_scorecard.py (core generator) · src/gaia/eval/scorecard_gate.py (release gate)Distinct from src/gaia/eval/scorecard.py — that file is the per-run scenario PASS/FAIL aggregator used internally by gaia eval agent. This document describes the outward-facing release artifact.

Overview

Each published hub agent ships a release scorecard — a single SCORECARD.md file (updated in place per release, versioned via the publish snapshot, the same way README.md works) that records:

The eval recipe: dataset reference, methodology, configuration, and metric definitions.
The measured results: per-metric values, number of test cases actually run, and dataset size.
A single named aggregate score: a deterministic, recomputable percentage so a reviewer can verify the number without re-running the eval.
A Reproduction section: the exact commands to reproduce the result from scratch.

Scorecards are committed alongside the agent’s README and linked from it. A standalone release gate (scorecard_gate.py) blocks packaging when the scorecard is missing, when its aggregate score regresses below the prior version’s, or — when enforcement is on — when it is below an absolute bar or an anti-gaming floor.

Acceptance metric — email triage (#1437)

The email-triage scorecard’s aggregate is within-one-bucket acceptance accuracy, not exact 4-way match. Triage priority is an ordinal scale, so the user-facing question is “how far off”, not “exact match”:

URGENT (3) > NEEDS_RESPONSE (2) > FYI (1) > PROMOTIONAL (0)

A prediction is credited when it is exact or an adjacent bucket (|rank(pred) − rank(expected)| ≤ 1) — what a user feels (nothing urgent buried in low-priority). PERSONAL is not a priority level and is absent from the corpus, so it is scored exact-only; re-place it on the scale if the corpus gains PERSONAL examples. Exact 4-way agreement (~40% on a 4B on-device model) is below the realistic ceiling — single-rater human agreement on subjective priority is only ~60–75% — so the 80% bar (#1437) is on this acceptance metric, with these reported (non-gating) secondaries:

urgent_vs_not_accuracy — binary accuracy on the needs-attention axis ({URGENT, NEEDS_RESPONSE} vs rest).
urgent_recall — recall on that axis; the input to the gate’s anti-gaming URGENT floor.
category_accuracy — exact 4-way match, kept as a reference.

The secondaries are recorded with weight 0, so they are displayed but excluded from aggregate.value (which stays round(100 × within_one_bucket_accuracy, 2), recomputable from the displayed components per the formula below).

Trustworthy numbers — variance & CI (#1894)

The corpus + greedy decoding (temperature=0.0) are fixed, and the needs_llm routing is a deterministic heuristic, so the only run-to-run noise is GPU floating-point non-determinism (not seedable — measured, not removed). gaia eval benchmark --experiments N runs N times over the same corpus and the adapter records an additive acceptance_variance block (mean / stdev / CV% / 95% CI / n_runs) in recipe.config. It never affects aggregate.value (the mean); it lets the gate tell a real regression from noise.

File format

Scorecard files are Markdown with YAML front matter (.md). The front matter holds all machine-readable fields; the body is a human-readable summary with a worked recomputation and a Reproduction section.

---
schema_version: 1
agent:
  name: Email Triage
  version: 0.3.0
recipe:
  dataset:
    reference: tests/fixtures/email/ground_truth.json
    description: Synthetic email corpus (FakeGmailBackend, schema-2.0 triage taxonomy)
    size: 220
  methodology: gaia eval benchmark — within-one-bucket acceptance (exact-or-adjacent, #1437)
  config:
    harness: gaia eval benchmark
    model: Gemma-4-E4B-it-GGUF
    limit: 220
    n_runs: 3
    # acceptance_variance: { ... } — additive mean/stdev/CI per metric (omitted here)
results:
  test_cases_run: 100
  metrics:
    - name: within_one_bucket_accuracy   # gated aggregate
      value: 0.8467
      weight: 1.0
    - name: urgent_vs_not_accuracy       # reported secondaries (weight 0)
      value: 0.58
      weight: 0.0
    - name: urgent_recall
      value: 0.6571
      weight: 0.0
    - name: category_accuracy
      value: 0.4567
      weight: 0.0
aggregate:
  name: weighted_accuracy
  formula: "round(100 * sum(weight_i * value_i) / sum(weight_i), 2)"
  components:
    - metric: within_one_bucket_accuracy
      value: 0.8467
      weight: 1.0
    # ... weight-0 secondaries omitted for brevity
  value: 84.67
generated_at: "2026-06-29T21:45:03+00:00"
inherited_from: null
---

# Email Triage — Eval Scorecard v0.3.0

**Aggregate score: 84.67** (out of 100)
...

## Reproduction

Run the following commands from the repository root:
...

Required fields

A scorecard missing any of these is invalid and will be rejected by the release gate:

Field	Description
`schema_version`	Always `1` for this schema version
`agent.name`	Human-readable agent name
`agent.version`	Semver version string (e.g. `0.2.4`)
`recipe.dataset.reference`	Dataset path or URL
`recipe.dataset.description`	Short description
`recipe.dataset.size`	Total labeled examples available
`recipe.methodology`	How the eval was run
`recipe.config`	Harness config (model, limit, corpus, …)
`results.test_cases_run`	Subset of examples actually executed this run
`results.metrics`	List of `{name, value, weight}` dicts
`aggregate.name`	Name of the aggregate score
`aggregate.formula`	Human-readable formula string
`aggregate.components`	List of `{metric, value, weight}` dicts
`aggregate.value`	The computed aggregate float

Optional fields

These blocks are optional and additive — a scorecard is valid with or without them, and none of them ever affects aggregate.value. They add reviewer-facing detail about how the number was produced and where the misses are:

Field	Description
`recipe.environment`	What produced the numbers: `gaia_commit`, `lemonade_version`, `model`, `hardware` (a class descriptor — never a hostname), and an optional `temperature`.
`results.breakdown`	Where the score comes from: `per_category` (per-label `total` / `correct` / `accuracy`) and `top_confusions` (the most frequent `expected → predicted` mistakes).
`results.performance`	Observed perf figures, mean across the run’s scenarios: `ttft_s`, `throughput_tps`, `pipeline_s`, `peak_memory_gb`, `emails_per_run`, and the token-accounting rows below. Rendered as the `## Performance` table. Every row here is reported, never gated.

Token accounting rows (results.performance, issue #1891) — real LLM token counts including the nested per-email classify calls the outer-turn stats used to miss:

Row	Definition
`total_input_tokens` / `total_output_tokens`	Run totals: outer agent-loop turns plus the triage tool’s per-email classify calls.
`llm_classified_count`	Classify calls whose usage was measurable — on the shipped Lemonade path this equals the number of emails classified by the LLM (the rest took the heuristic fast path).
`tokens_per_triage`	Pinned definition: triage classify-call tokens ÷ `llm_classified_count` — classify tokens over LLM-classified emails only, so the figure is invariant to the heuristic/LLM mix and to outer-turn measurement races. Comparisons across releases are only meaningful with the corpus held fixed.

tokens_per_triage is reported, not gated: do not add a release-gate bar for it without a committed, ctx-stamped baseline scorecard to compare against (see _tokens_per_triage_comment in tests/fixtures/email/quality_gate_thresholds.json).

recipe:
  environment:
    gaia_commit: a1b2c3d
    lemonade_version: 10.8.0
    model: Gemma-4-E4B-it-GGUF
    hardware: AMD Ryzen AI MAX+ (Strix Halo)
results:
  breakdown:
    per_category:
      - {category: fyi, total: 88, correct: 54, accuracy: 0.6136}
      - {category: needs_response, total: 44, correct: 17, accuracy: 0.3864}
    top_confusions:
      - {expected: fyi, predicted: needs_response, count: 28}

The email-triage adapter fills both in automatically. For a new agent, set ResultPayload.environment / ResultPayload.breakdown when your harness exposes the data, and omit them otherwise.

Two counts — defined distinctly

recipe.dataset.size and results.test_cases_run are intentionally separate fields:

recipe.dataset.size — total labeled examples available in the dataset (fixed for a given dataset version).
results.test_cases_run — the subset actually executed in this run (may be limited by --limit). Must be ≤ recipe.dataset.size.

They may be numerically equal (when the full dataset is run), but they represent different things.

Comparability depends on a consistent --limit. Future regression checks compare aggregate scores. If one run uses --limit 12 and the next uses --limit 100, the scores may differ for reasons unrelated to model quality. Record the exact limit in recipe.config and keep it consistent across versions.

Aggregate score formula

aggregate.value = round(100 × Σ(weightᵢ × valueᵢ) / Σ(weightᵢ), 2)

where each valueᵢ is a metric value in [0, 1] and each weightᵢ defaults to 1.0. The result is a percentage in [0, 100]. For a single metric with weight 1.0:

round(100 × 0.40, 2) = 40.0

A reader can reproduce this value from aggregate.components alone — no eval-harness access needed. The aggregate.formula field in the front matter states the formula in human-readable form so it is self-documenting.

Storage convention

Each agent package ships a single SCORECARD.md file, updated in place per release — the same way README.md works. Per-version uniqueness comes from the publish snapshot (R2 stores the file at agents/<id>/<version>/SCORECARD.md; the npm package ships only the current version’s SCORECARD.md).

<doc-root>/
  README.md              ← canonical README (links to SCORECARD.md)
  SCORECARD.md           ← current version's scorecard, updated in place
  SPEC.md
  SKILL.md
  CHANGELOG.md

The doc-root is the location of the agent’s canonical README:

Agent	doc-root
Email Triage (`@amd-gaia/agent-email`)	`hub/agents/npm/agent-email/`

The relative link ./SCORECARD.md resolves both in-repo and when the directory is published as an npm package. The npm files array includes SCORECARD.md (not a scorecards/ directory).

Versioning policy

Patch releases — carry forward

For a patch release (same major.minor, patch incremented), the prior version’s results are carried forward verbatim using carry_forward(). Pass the path to the agent’s current SCORECARD.md:

from gaia.eval.release_scorecard import carry_forward, write_scorecard
from pathlib import Path

new_payload = carry_forward(
    prev_scorecard_path=Path("hub/agents/npm/agent-email/SCORECARD.md"),
    new_version="0.2.5",
)
# new_payload.inherited_from == "0.2.4"  (read from front matter, not filename)
write_scorecard(new_payload, Path("hub/agents/npm/agent-email/SCORECARD.md"))

The resulting scorecard has inherited_from: "0.2.4" and identical results and aggregate fields. The aggregate score is unchanged, so the release gate’s equal-score case passes. carry_forward() reads the prior version from the agent.version field in the front matter — not from the filename.

Minor / major releases — re-run required

For a minor or major bump, carry_forward() raises ValueError with a “re-run” message. Run the eval fresh and generate a new scorecard:

PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring \
GAIA_AGENT_TOOL_TIMEOUT=1800 \
PYTHONPATH="$(pwd)" \
gaia eval benchmark \
  --model Gemma-4-E4B-it-GGUF \
  --mbox-path tests/fixtures/email/synthetic_inbox.mbox \
  --ground-truth tests/fixtures/email/ground_truth.json \
  --limit 220 \
  --output-dir /tmp/email-eval

PYTHONPATH="$(pwd)" \
python hub/agents/python/email/packaging/gen_scorecard.py \
  --benchmark-dir /tmp/email-eval \
  --limit 220

This writes hub/agents/npm/agent-email/SCORECARD.md in place.

Release gate

scorecard_gate.py is a standalone script that exits non-zero on failure:

# Presence-only check (first adoption or no baseline specified):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/npm/agent-email/SCORECARD.md

# Regression check against a specific prior scorecard file (unit tests / local):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/npm/agent-email/SCORECARD.md \
  --baseline-file /tmp/prev-SCORECARD.md

# Regression check against a prior release tag (CI):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/npm/agent-email/SCORECARD.md \
  --baseline-ref agent-pkg-email-v0.2.3

--baseline-file and --baseline-ref are mutually exclusive. If the file doesn’t exist at the given ref, the gate treats it as first adoption (presence-only pass).

Gate logic

Presence check: --scorecard path must exist and be a valid scorecard. → exit 1 if not.
Absolute bar + anti-gaming floor (opt-in, applies to every path including first adoption):
- --min-aggregate <N>: exit 1 if aggregate.value < N (the #1437 80% bar is --min-aggregate 80).
- --min-urgent-recall <r>: exit 1 if the card’s urgent_recall secondary is below r (a high aggregate must not come with buried urgent mail). Fails loud if the metric is absent.
- Omit both for report mode (presence + regression only). The email release workflow reads these from the thresholds manifest (acceptance_target, urgent_recall_floor, acceptance_enforce) — data, not code.
Baseline resolution:
- --baseline-file: read the given file directly (no git access; suitable for unit tests).
- --baseline-ref: resolve via git show <ref>:<scorecard-path>. If the file does not exist at that ref → first adoption, exit 0.
- Neither specified: first adoption, exit 0 (presence-only pass).
Regression check (variance-aware, #1894): if the baseline records a within-one stdev, a regression is flagged only when candidate < baseline − k·stdev (--regression-k, default 1; stdev is on the [0,1] scale, scaled ×100 to match aggregate.value). With no recorded stdev, a strict < is used.
Equal or above the threshold → exit 0.

Exit codes

Case	Exit code
Missing or invalid candidate scorecard	`1`
Below `--min-aggregate` bar	`1`
Below `--min-urgent-recall` floor (or metric absent)	`1`
Regression beyond the variance band vs baseline	`1`
No baseline (first adoption), bar/floor met	`0`
File absent at `--baseline-ref`	`0`
Equal score (patch carry-forward)	`0`
Dip within the variance band, or score improved	`0`

`--allow-regression`

When a regression is intentional (e.g. a dataset correction or methodology change), use --allow-regression. The gate prints a GHA ::warning:: annotation naming both versions and scores, then exits 0:

::warning::Scorecard regression allowed by --allow-regression: v0.2.3=65.0 → v0.2.4=40.0
WARNING: Regression override active. Prior version v0.2.3 scored 65.0; candidate v0.2.4 scored 40.0. ...

Keeping the scorecard current (the update / reject loop)

The scorecard must move with the agent: when LLM-affecting code changes, the eval is re-run and the committed SCORECARD.md refreshed — upward. A regression is blocked. Two enforcement points work together:

Reject-on-worse (always on, GitHub-hosted). The scorecard-gate job in release_agent_<id>.yml runs on every release. It only parses committed files (no eval), so it runs on a standard runner and fails the build if the committed scorecard regressed below the prior version or is missing. This is the hard gate.
Run-and-refresh (self-hosted AMD, manual dispatch). gaia eval benchmark needs Lemonade on AMD hardware, so it cannot run on GitHub-hosted runners — it runs on the [self-hosted, Windows, stx] pool, serialized against the other evals by the shared lemonade-eval concurrency group. The Email Agent Eval — scorecard refresh workflow (.github/workflows/email_scorecard_refresh.yml) is workflow_dispatch-only (#2094 — it previously also ran on pushes touching the email agent, but a full eval costs ~10.5h and that trigger never once completed a run), and takes two profiles:
- full (limit ≥ 249 and experiments ≥ the committed card’s n_runs) — reproduces the committed card’s methodology and is the only profile allowed to commit a refreshed card;
- subset (limit < 249) — an end-to-end smoke run that exercises every step and then never commits (a small-sample number is not a valid stand-in for the published figure).
A full run then either commits the refreshed card (score clears the absolute bar and shows no regression beyond the recorded noise band) or fails loudly (regression, or a card below the acceptance bar / urgent-recall floor from tests/fixtures/email/quality_gate_thresholds.json). Before any eval spend, the run also fails fast if its basis (ctx_size, experiments) differs from the committed card’s — unless rebaseline=true (with a required rebaseline_reason) marks the basis change deliberate.

So the published number always traces to a full-corpus run, and the release gate is the backstop on hosted CI. Locally, gen_scorecard.py + scorecard_gate.py reproduce both steps (see the adding-eval-scorecard skill).

The refresh job needs contents: write and runs only on the repo’s own branches — a fork PR’s GITHUB_TOKEN is read-only and cannot auto-commit. For a fork PR, run the eval locally/on AMD hardware and commit the scorecard manually; the release gate still enforces no-regression.

Adding a scorecard for a new agent

Use the adding-eval-scorecard skill. In Claude Code, invoke it instead of following these steps by hand — it carries the exact commands, the harness→payload→generator flow, the headless-eval gotchas (keyring/PYTHONPATH/tool-timeout), and the verification evidence to capture. The steps below are the reference the skill automates.

Write a packaging/gen_scorecard.py adapter (see hub/agents/python/email/packaging/gen_scorecard.py for a reference). The adapter should populate reproduction_command with the exact commands needed to reproduce the scorecard.
Run the eval and call the adapter → commit the resulting SCORECARD.md to <doc-root>/SCORECARD.md.
Link the scorecard from the README: ./SCORECARD.md.
Add SCORECARD.md to the npm package.json files array (if published on npm); do not add a scorecards/ directory.
Wire scorecard_gate into the release workflow (see release_agent_email.yml for the job topology). Use --scorecard <path>/SCORECARD.md and --baseline-ref <prev-tag> (best-effort).

Command Line

API & Development

Help

Release Eval Scorecard

Overview

Acceptance metric — email triage (#1437)

Trustworthy numbers — variance & CI (#1894)

File format

Required fields

Optional fields

Two counts — defined distinctly

Aggregate score formula

Storage convention

Versioning policy

Patch releases — carry forward

Minor / major releases — re-run required

Release gate

Gate logic

Exit codes

`--allow-regression`

Keeping the scorecard current (the update / reject loop)

Adding a scorecard for a new agent

​Overview

​Acceptance metric — email triage (#1437)

​Trustworthy numbers — variance & CI (#1894)

​File format

​Required fields

​Optional fields

​Two counts — defined distinctly

​Aggregate score formula

​Storage convention

​Versioning policy

​Patch releases — carry forward

​Minor / major releases — re-run required

​Release gate

​Gate logic

​Exit codes

​--allow-regression

​Keeping the scorecard current (the update / reject loop)

​Adding a scorecard for a new agent

Overview

Acceptance metric — email triage (#1437)

Trustworthy numbers — variance & CI (#1894)

File format

Required fields

Optional fields

Two counts — defined distinctly

Aggregate score formula

Storage convention

Versioning policy

Patch releases — carry forward

Minor / major releases — re-run required

Release gate

Gate logic

Exit codes

`--allow-regression`

Keeping the scorecard current (the update / reject loop)

Adding a scorecard for a new agent