> ## Documentation Index
> Fetch the complete documentation index at: https://amd-gaia.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Release Eval Scorecard

> Per-agent eval scorecard: schema, storage convention, aggregate formula, versioning policy, reproduction, and release gate.

<Info>
  **Source Code:**
  [`src/gaia/eval/release_scorecard.py`](https://github.com/amd/gaia/tree/main/src/gaia/eval/release_scorecard.py) (core generator) ·
  [`src/gaia/eval/scorecard_gate.py`](https://github.com/amd/gaia/tree/main/src/gaia/eval/scorecard_gate.py) (release gate)

  **Distinct from** [`src/gaia/eval/scorecard.py`](https://github.com/amd/gaia/tree/main/src/gaia/eval/scorecard.py) — that file is the per-run scenario PASS/FAIL aggregator used internally by `gaia eval agent`. This document describes the outward-facing *release artifact*.
</Info>

## Overview

Each published hub agent ships a **release scorecard** — a single `SCORECARD.md` file (updated in place per release, versioned via the publish snapshot, the same way `README.md` works) that records:

* The **eval recipe**: dataset reference, methodology, configuration, and metric definitions.
* The **measured results**: per-metric values, number of test cases actually run, and dataset size.
* A single **named aggregate score**: a deterministic, recomputable percentage so a reviewer can verify the number without re-running the eval.
* A **Reproduction section**: the exact commands to reproduce the result from scratch.

Scorecards are committed alongside the agent's README and linked from it. A standalone **release gate** (`scorecard_gate.py`) blocks packaging when the scorecard is missing, when its aggregate score regresses below the prior version's, or — when enforcement is on — when it is below an absolute bar or an anti-gaming floor.

## Acceptance metric — email triage (#1437)

The email-triage scorecard's aggregate is **within-one-bucket acceptance accuracy**, not exact 4-way match. Triage priority is an *ordinal* scale, so the user-facing question is "how far off", not "exact match":

```
URGENT (3) > NEEDS_RESPONSE (2) > FYI (1) > PROMOTIONAL (0)
```

A prediction is credited when it is exact **or an adjacent bucket** (`|rank(pred) − rank(expected)| ≤ 1`) — what a user feels (nothing urgent buried in low-priority). `PERSONAL` is not a priority *level* and is absent from the corpus, so it is scored exact-only; re-place it on the scale if the corpus gains `PERSONAL` examples. Exact 4-way agreement (\~40% on a 4B on-device model) is below the realistic ceiling — single-rater human agreement on subjective priority is only \~60–75% — so the **80% bar (#1437) is on this acceptance metric**, with these **reported (non-gating) secondaries**:

* `urgent_vs_not_accuracy` — binary accuracy on the needs-attention axis (`{URGENT, NEEDS_RESPONSE}` vs rest).
* `urgent_recall` — recall on that axis; the input to the gate's anti-gaming URGENT floor.
* `category_accuracy` — exact 4-way match, kept as a reference.

The secondaries are recorded with **weight 0**, so they are displayed but excluded from `aggregate.value` (which stays `round(100 × within_one_bucket_accuracy, 2)`, recomputable from the displayed components per the formula below).

### Trustworthy numbers — variance & CI (#1894)

The corpus + greedy decoding (`temperature=0.0`) are fixed, and the `needs_llm` routing is a deterministic heuristic, so the only run-to-run noise is GPU floating-point non-determinism (not seedable — measured, not removed). `gaia eval benchmark --experiments N` runs N times over the same corpus and the adapter records an additive `acceptance_variance` block (mean / stdev / CV% / 95% CI / `n_runs`) in `recipe.config`. It never affects `aggregate.value` (the mean); it lets the gate tell a real regression from noise.

## File format

Scorecard files are **Markdown with YAML front matter** (`.md`). The front matter holds all machine-readable fields; the body is a human-readable summary with a worked recomputation and a Reproduction section.

```
---
schema_version: 1
agent:
  name: Email Triage
  version: 0.3.0
recipe:
  dataset:
    reference: tests/fixtures/email/ground_truth.json
    description: Synthetic email corpus (FakeGmailBackend, schema-2.0 triage taxonomy)
    size: 220
  methodology: gaia eval benchmark — within-one-bucket acceptance (exact-or-adjacent, #1437)
  config:
    harness: gaia eval benchmark
    model: Gemma-4-E4B-it-GGUF
    limit: 220
    n_runs: 3
    # acceptance_variance: { ... } — additive mean/stdev/CI per metric (omitted here)
results:
  test_cases_run: 100
  metrics:
    - name: within_one_bucket_accuracy   # gated aggregate
      value: 0.8467
      weight: 1.0
    - name: urgent_vs_not_accuracy       # reported secondaries (weight 0)
      value: 0.58
      weight: 0.0
    - name: urgent_recall
      value: 0.6571
      weight: 0.0
    - name: category_accuracy
      value: 0.4567
      weight: 0.0
aggregate:
  name: weighted_accuracy
  formula: "round(100 * sum(weight_i * value_i) / sum(weight_i), 2)"
  components:
    - metric: within_one_bucket_accuracy
      value: 0.8467
      weight: 1.0
    # ... weight-0 secondaries omitted for brevity
  value: 84.67
generated_at: "2026-06-29T21:45:03+00:00"
inherited_from: null
---

# Email Triage — Eval Scorecard v0.3.0

**Aggregate score: 84.67** (out of 100)
...

## Reproduction

Run the following commands from the repository root:
...
```

### Required fields

A scorecard missing any of these is **invalid** and will be rejected by the release gate:

| Field                        | Description                                   |
| ---------------------------- | --------------------------------------------- |
| `schema_version`             | Always `1` for this schema version            |
| `agent.name`                 | Human-readable agent name                     |
| `agent.version`              | Semver version string (e.g. `0.2.4`)          |
| `recipe.dataset.reference`   | Dataset path or URL                           |
| `recipe.dataset.description` | Short description                             |
| `recipe.dataset.size`        | Total labeled examples available              |
| `recipe.methodology`         | How the eval was run                          |
| `recipe.config`              | Harness config (model, limit, corpus, …)      |
| `results.test_cases_run`     | Subset of examples actually executed this run |
| `results.metrics`            | List of `{name, value, weight}` dicts         |
| `aggregate.name`             | Name of the aggregate score                   |
| `aggregate.formula`          | Human-readable formula string                 |
| `aggregate.components`       | List of `{metric, value, weight}` dicts       |
| `aggregate.value`            | The computed aggregate float                  |

### Optional fields

These blocks are **optional and additive** — a scorecard is valid with or without them, and none of them ever affects `aggregate.value`. They add reviewer-facing detail about *how* the number was produced and *where* the misses are:

| Field                 | Description                                                                                                                                                                                                                                                     |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `recipe.environment`  | What produced the numbers: `gaia_commit`, `lemonade_version`, `model`, `hardware` (a class descriptor — never a hostname), and an optional `temperature`.                                                                                                       |
| `results.breakdown`   | Where the score comes from: `per_category` (per-label `total` / `correct` / `accuracy`) and `top_confusions` (the most frequent `expected → predicted` mistakes).                                                                                               |
| `results.performance` | Observed perf figures, mean across the run's scenarios: `ttft_s`, `throughput_tps`, `pipeline_s`, `peak_memory_gb`, `emails_per_run`, and the token-accounting rows below. Rendered as the `## Performance` table. Every row here is **reported, never gated**. |

**Token accounting rows** (`results.performance`, issue #1891) — real LLM token counts including the nested per-email classify calls the outer-turn stats used to miss:

| Row                                          | Definition                                                                                                                                                                                                                                                                                            |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `total_input_tokens` / `total_output_tokens` | Run totals: outer agent-loop turns **plus** the triage tool's per-email classify calls.                                                                                                                                                                                                               |
| `llm_classified_count`                       | Classify calls whose usage was measurable — on the shipped Lemonade path this equals the number of emails classified by the LLM (the rest took the heuristic fast path).                                                                                                                              |
| `tokens_per_triage`                          | **Pinned definition:** triage classify-call tokens ÷ `llm_classified_count` — classify tokens over *LLM-classified emails only*, so the figure is invariant to the heuristic/LLM mix and to outer-turn measurement races. Comparisons across releases are only meaningful with the corpus held fixed. |

<Note>
  `tokens_per_triage` is **reported, not gated**: do not add a release-gate bar for it without a committed, ctx-stamped baseline scorecard to compare against (see `_tokens_per_triage_comment` in `tests/fixtures/email/quality_gate_thresholds.json`).
</Note>

```yaml theme={null}
recipe:
  environment:
    gaia_commit: a1b2c3d
    lemonade_version: 10.8.0
    model: Gemma-4-E4B-it-GGUF
    hardware: AMD Ryzen AI MAX+ (Strix Halo)
results:
  breakdown:
    per_category:
      - {category: fyi, total: 88, correct: 54, accuracy: 0.6136}
      - {category: needs_response, total: 44, correct: 17, accuracy: 0.3864}
    top_confusions:
      - {expected: fyi, predicted: needs_response, count: 28}
```

The email-triage adapter fills both in automatically. For a new agent, set `ResultPayload.environment` / `ResultPayload.breakdown` when your harness exposes the data, and omit them otherwise.

### Two counts — defined distinctly

`recipe.dataset.size` and `results.test_cases_run` are intentionally **separate fields**:

* **`recipe.dataset.size`** — total labeled examples available in the dataset (fixed for a given dataset version).
* **`results.test_cases_run`** — the subset actually executed in this run (may be limited by `--limit`). Must be ≤ `recipe.dataset.size`.

They may be numerically equal (when the full dataset is run), but they represent different things.

<Warning>
  **Comparability depends on a consistent `--limit`.** Future regression checks compare aggregate scores. If one run uses `--limit 12` and the next uses `--limit 100`, the scores may differ for reasons unrelated to model quality. Record the exact `limit` in `recipe.config` and keep it consistent across versions.
</Warning>

## Aggregate score formula

```
aggregate.value = round(100 × Σ(weightᵢ × valueᵢ) / Σ(weightᵢ), 2)
```

where each `valueᵢ` is a metric value in \[0, 1] and each `weightᵢ` defaults to 1.0.

The result is a **percentage in \[0, 100]**. For a single metric with weight 1.0:

```
round(100 × 0.40, 2) = 40.0
```

A reader can reproduce this value from `aggregate.components` alone — no eval-harness access needed.
The `aggregate.formula` field in the front matter states the formula in human-readable form so it is self-documenting.

## Storage convention

Each agent package ships a **single `SCORECARD.md`** file, updated in place per release — the same way `README.md` works. Per-version uniqueness comes from the publish snapshot (R2 stores the file at `agents/<id>/<version>/SCORECARD.md`; the npm package ships only the current version's `SCORECARD.md`).

```
<doc-root>/
  README.md              ← canonical README (links to SCORECARD.md)
  SCORECARD.md           ← current version's scorecard, updated in place
  SPEC.md
  SKILL.md
  CHANGELOG.md
```

The `doc-root` is the location of the agent's canonical README:

| Agent                                  | doc-root                |
| -------------------------------------- | ----------------------- |
| Email Triage (`@amd-gaia/agent-email`) | `hub/agents/email/npm/` |

The relative link `./SCORECARD.md` resolves both in-repo and when the directory is published as an npm package. The npm `files` array includes `SCORECARD.md` (not a `scorecards/` directory).

## Versioning policy

### Patch releases — carry forward

For a **patch release** (same `major.minor`, `patch` incremented), the prior version's results are carried forward verbatim using `carry_forward()`. Pass the path to the agent's current `SCORECARD.md`:

```python theme={null}
from gaia.eval.release_scorecard import carry_forward, write_scorecard
from pathlib import Path

new_payload = carry_forward(
    prev_scorecard_path=Path("hub/agents/email/npm/SCORECARD.md"),
    new_version="0.2.5",
)
# new_payload.inherited_from == "0.2.4"  (read from front matter, not filename)
write_scorecard(new_payload, Path("hub/agents/email/npm/SCORECARD.md"))
```

The resulting scorecard has `inherited_from: "0.2.4"` and identical `results` and `aggregate` fields. The aggregate score is unchanged, so the release gate's equal-score case passes.

`carry_forward()` reads the prior version from the `agent.version` field in the front matter — **not** from the filename.

### Minor / major releases — re-run required

For a **minor or major bump**, `carry_forward()` raises `ValueError` with a "re-run" message. Run the eval fresh and generate a new scorecard:

```bash theme={null}
PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring \
GAIA_AGENT_TOOL_TIMEOUT=1800 \
PYTHONPATH="$(pwd)" \
gaia eval benchmark \
  --model Gemma-4-E4B-it-GGUF \
  --mbox-path tests/fixtures/email/synthetic_inbox.mbox \
  --ground-truth tests/fixtures/email/ground_truth.json \
  --limit 220 \
  --output-dir /tmp/email-eval

PYTHONPATH="$(pwd)" \
python hub/agents/email/python/packaging/gen_scorecard.py \
  --benchmark-dir /tmp/email-eval \
  --limit 220
```

This writes `hub/agents/email/npm/SCORECARD.md` in place.

## Release gate

`scorecard_gate.py` is a standalone script that exits non-zero on failure:

```bash theme={null}
# Presence-only check (first adoption or no baseline specified):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/email/npm/SCORECARD.md

# Regression check against a specific prior scorecard file (unit tests / local):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/email/npm/SCORECARD.md \
  --baseline-file /tmp/prev-SCORECARD.md

# Regression check against a prior release tag (CI):
python -m gaia.eval.scorecard_gate \
  --scorecard hub/agents/email/npm/SCORECARD.md \
  --baseline-ref agent-pkg-email-v0.2.3
```

`--baseline-file` and `--baseline-ref` are mutually exclusive. If the file doesn't exist at the given ref, the gate treats it as first adoption (presence-only pass).

### Gate logic

1. **Presence check**: `--scorecard` path must exist and be a valid scorecard. → exit 1 if not.
2. **Absolute bar + anti-gaming floor** (opt-in, applies to *every* path including first adoption):
   * `--min-aggregate <N>`: exit 1 if `aggregate.value < N` (the #1437 80% bar is `--min-aggregate 80`).
   * `--min-urgent-recall <r>`: exit 1 if the card's `urgent_recall` secondary is below `r` (a high aggregate must not come with buried urgent mail). Fails loud if the metric is absent.
   * Omit both for report mode (presence + regression only). The email release workflow reads these from the thresholds manifest (`acceptance_target`, `urgent_recall_floor`, `acceptance_enforce`) — data, not code.
3. **Baseline resolution**:
   * `--baseline-file`: read the given file directly (no git access; suitable for unit tests).
   * `--baseline-ref`: resolve via `git show <ref>:<scorecard-path>`. If the file does not exist at that ref → **first adoption**, exit 0.
   * Neither specified: **first adoption**, exit 0 (presence-only pass).
4. **Regression check** (variance-aware, #1894): if the baseline records a within-one stdev, a regression is flagged only when `candidate < baseline − k·stdev` (`--regression-k`, default 1; stdev is on the \[0,1] scale, scaled ×100 to match `aggregate.value`). With no recorded stdev, a strict `<` is used.
5. Equal or above the threshold → exit 0.

### Exit codes

| Case                                                 | Exit code |
| ---------------------------------------------------- | --------- |
| Missing or invalid candidate scorecard               | `1`       |
| Below `--min-aggregate` bar                          | `1`       |
| Below `--min-urgent-recall` floor (or metric absent) | `1`       |
| Regression beyond the variance band vs baseline      | `1`       |
| No baseline (first adoption), bar/floor met          | `0`       |
| File absent at `--baseline-ref`                      | `0`       |
| Equal score (patch carry-forward)                    | `0`       |
| Dip within the variance band, or score improved      | `0`       |

### `--allow-regression`

When a regression is intentional (e.g. a dataset correction or methodology change), use `--allow-regression`. The gate prints a GHA `::warning::` annotation naming both versions and scores, then exits 0:

```
::warning::Scorecard regression allowed by --allow-regression: v0.2.3=65.0 → v0.2.4=40.0
WARNING: Regression override active. Prior version v0.2.3 scored 65.0; candidate v0.2.4 scored 40.0. ...
```

## Keeping the scorecard current (the update / reject loop)

The scorecard must move with the agent: when LLM-affecting code changes, the eval is re-run and the committed `SCORECARD.md` refreshed — **upward**. A regression is blocked.

Two enforcement points work together:

1. **Reject-on-worse (always on, GitHub-hosted).** The `scorecard-gate` job in `release_agent_<id>.yml` runs on every release. It only parses committed files (no eval), so it runs on a standard runner and **fails the build** if the committed scorecard regressed below the prior version or is missing. This is the hard gate.
2. **Run-and-refresh (self-hosted AMD, manual dispatch).** `gaia eval benchmark` needs Lemonade on AMD hardware, so it cannot run on GitHub-hosted runners — it runs on the `[self-hosted, Windows, stx]` pool, serialized against the other evals by the shared `lemonade-eval` concurrency group. The `Email Agent Eval — scorecard refresh` workflow (`.github/workflows/email_scorecard_refresh.yml`) is **`workflow_dispatch`-only** (#2094 — it previously also ran on pushes touching the email agent, but a full eval costs \~10.5h and that trigger never once completed a run), and takes two profiles:

   * **full** (`limit ≥ 249` **and** `experiments ≥` the committed card's `n_runs`) — reproduces the committed card's methodology and is the **only** profile allowed to commit a refreshed card;
   * **subset** (`limit < 249`) — an end-to-end smoke run that exercises every step and then **never commits** (a small-sample number is not a valid stand-in for the published figure).

   A full run then either **commits** the refreshed card (score clears the absolute bar and shows no regression beyond the recorded noise band) or **fails loudly** (regression, or a card below the acceptance bar / urgent-recall floor from `tests/fixtures/email/quality_gate_thresholds.json`). Before any eval spend, the run also fails fast if its basis (`ctx_size`, `experiments`) differs from the committed card's — unless `rebaseline=true` (with a required `rebaseline_reason`) marks the basis change deliberate.

So the published number always traces to a full-corpus run, and the release gate is the backstop on hosted CI. Locally, `gen_scorecard.py` + `scorecard_gate.py` reproduce both steps (see the **`adding-eval-scorecard` skill**).

<Warning>
  The refresh job needs `contents: write` and runs only on the repo's own branches — a fork PR's `GITHUB_TOKEN` is read-only and cannot auto-commit. For a fork PR, run the eval locally/on AMD hardware and commit the scorecard manually; the release gate still enforces no-regression.
</Warning>

## Adding a scorecard for a new agent

<Tip>
  **Use the [`adding-eval-scorecard` skill](https://github.com/amd/gaia/tree/main/.claude/skills/adding-eval-scorecard/SKILL.md).** In Claude Code, invoke it instead of following these steps by hand — it carries the exact commands, the harness→payload→generator flow, the headless-eval gotchas (keyring/PYTHONPATH/tool-timeout), and the verification evidence to capture. The steps below are the reference the skill automates.
</Tip>

1. Write a `packaging/gen_scorecard.py` adapter (see `hub/agents/email/python/packaging/gen_scorecard.py` for a reference). The adapter should populate `reproduction_command` with the exact commands needed to reproduce the scorecard.
2. Run the eval and call the adapter → commit the resulting `SCORECARD.md` to `<doc-root>/SCORECARD.md`.
3. Link the scorecard from the README: `./SCORECARD.md`.
4. Add `SCORECARD.md` to the npm `package.json` `files` array (if published on npm); do **not** add a `scorecards/` directory.
5. Wire `scorecard_gate` into the release workflow (see `release_agent_email.yml` for the job topology). Use `--scorecard <path>/SCORECARD.md` and `--baseline-ref <prev-tag>` (best-effort).