> ## Documentation Index
> Fetch the complete documentation index at: https://amd-gaia.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Benchmarks

> Compare GAIA's agent against Claude Code, or one model against another, on the same fixed tasks

> [!NOTE]
>
> **Source Code:** [`src/gaia/eval/bench/`](https://github.com/amd/gaia/tree/main/src/gaia/eval/bench),
> [`src/gaia/eval/flagship_tasks.py`](https://github.com/amd/gaia/blob/main/src/gaia/eval/flagship_tasks.py),
> [`eval/tasks/tasks.json`](https://github.com/amd/gaia/blob/main/eval/tasks/tasks.json)

## What this is

`gaia eval tasks` gives an autonomous coding agent real work — fix a bug, triage
issues, write a weekly report from a CSV — then checks what actually happened: did
the work get done, was the agent honest about it, and what did it cost. The same
fixed set of tasks, time limit, tools and judge run against **two harnesses** — the
flagship GaiaAgent, or Claude Code driven headlessly — so a difference in the numbers
is a difference between agents, not between conditions.

This is the tool behind claims like "GLM-5.3 Flash under GAIA scores within about a
tenth of a point of Claude Opus 5 under Claude Code, for a fraction of the cost."
It answers them with a reproducible run anyone can repeat, not a one-off transcript
nobody else can check.

> [!WARNING]
>
> This drives a real agent against real models — it spends real tokens and, on a
> metered provider, real money. It is a contributor/maintainer tool for comparing
> harnesses and models, not something an end user runs.

## Prerequisites

* The flagship agent installed: `pip install -e hub/agents/gaia/python` (see the
  [Dev Guide](https://amd-gaia.ai/docs/reference/dev.md) if you haven't set up the base install).
* A running **Lemonade Server** with a cloud provider registered — Fireworks, for
  the open-weights models these benchmarks target. `gaia eval tasks` reaches every
  model through Lemonade's OpenAI-compatible endpoint.
* The **Claude Code CLI** (`claude`) on `PATH`, signed in to a subscription. It does
  two jobs here, both riding the subscription rather than an API key: it drives the
  `--harness claude-code` leg of a comparison, and it is **always** the judge, no
  matter which harness ran the task.

Nothing here needs `ANTHROPIC_API_KEY` — if `claude` already works on its own,
you're set.

## Quick start

> [!WARNING]
>
> The default output directory, `eval/results/eval-tasks-<timestamp>`, is
> gitignored, but only by its `eval-*` name: an `--out` elsewhere in the repo is
> not. Point `--out` (or `$GAIA_BENCH_RESULTS_DIR`) outside the repo, as below,
> and never commit anything a run writes.

```bash theme={null}
BENCH=~/gaia-bench
mkdir -p "$BENCH"
```

**1. Check the judge, before trusting any quality number**

Plants three attempts at one real task — a correct fix with its test suite
actually run, the same fix with an invented test run, and no work at all
claiming to be done — and requires the judge to grade them apart. Takes under a
minute.

```bash theme={null}
gaia eval tasks controls --out "$BENCH/controls"
```

**2. Run one task under GAIA's own harness**

`--tasks` narrows a suite to specific task ids, so a first try doesn't spend the
full 14-task run. This one needs the judge (it's a question task with no
mechanical check), so don't add `--no-judge`.

```bash theme={null}
gaia eval tasks run --suite everyday --tasks 21-qa \
  --harness gaia --model fireworks.glm-5p3-flash \
  --out "$BENCH/first-run"
```

Writes `scorecard.json`, plus `21-qa/transcript.json` and
`21-qa/workspace.diff`, under `$BENCH/first-run`.

**3. Render the report**

Builds the harness × model table from one or more finished runs. With a single
run directory it's a one-row table — useful to confirm the pipeline end to end
before running a real comparison.

```bash theme={null}
gaia eval tasks report "$BENCH/first-run" --out "$BENCH/first-report"
```

Writes `report.md`, `report.html`, and `report.png` where a headless Chrome is
on the machine.

## Harness vs. model

A **model** is the LLM doing the thinking — GLM-5.3 Flash, Claude Opus 5, Gemma. A
**harness** is the agent loop wrapped around it: how it reads the task, decides which
tool to call, and decides when it's done. GAIA has its own harness (the flagship
`GaiaAgent`); Claude Code is a different one. The same model can score differently
under each, because the harness — not just the model — decides how well that
thinking turns into finished work.

`--harness gaia|claude-code` is what isolates that: same tasks, same time limit,
same judge, same toolchain on `PATH`, only the harness changes. Running the same
model through both legs is how you find out whether a weak score is the model or the
wrapper around it.

## The three benchmarks

| Suite | What it is | Scored by |
| - | - | - |
| `everyday` | 14 small coding and GitHub tasks on `eval/tasks/toybox`, a synthetic Python project built for this: explain a file, fix a bug with a regression test, triage GitHub issues, find a duplicate, draft a reply, refactor, review a patch, write tests, a weekly CSV report, a shell-injection fix, label issues, config from an env var. The GitHub tasks run against an offline `gh` stand-in serving committed fixtures — no task reaches a personal login. | Running the code, plus the judge |
| `therock` | 4 real bugs from the public [ROCm/TheRock](https://github.com/ROCm/TheRock) repository, each checked out at the commit **before** its fix. The agent's diff is graded against the pull request that actually fixed it, fetched only after the agent has exited. Two of the four forbid internet use in the prompt. | The judge, against the real fix |
| `swebench` | Instances of [SWE-bench Verified](https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified), the public benchmark of real GitHub issues: the agent gets the repository at the commit before the fix and the issue text, and its patch is graded by the official SWE-bench harness running the repository's own tests in Docker. Nothing of the dataset is in this repo; see [SWE-bench Verified](#swe-bench-verified) below. | The official harness (the judge grades quality only) |

`everyday` is the one to reach for first — small, fast, cheap, and mechanically
checkable for most tasks. `therock` is a real 144K-line codebase with a real fix as
the answer key; it costs more and takes longer, and it's what proves the agent's
skills transfer past a project built for the benchmark. `swebench` is the public
yardstick: the same instances, prompt discipline and grader everyone else reports
against, so a number from it can be set beside a published one.

## How scoring works

Every `everyday` task starts from a fresh copy of `toybox`; the agent may read and
write that copy and nothing else. Scoring runs in two stages:

**Mechanical checks first**, for tasks where the project itself can decide:

* `tests_pass` — the project's own test suite passes.
* `probe` — a short snippet run inside the finished project, checking the actual
  behavior (e.g. the bug fix now accepts a lowercase input the original rejected).
* `unchanged` — named files are byte-identical to the pre-run snapshot, for tasks
  where touching them is itself a failure (a code-review task that edits the file
  under review, say).
* **Verified** additionally checks the agent's own tool record for a `pytest` run
  that passed after its last file edit — proof the "4 tests pass" in the answer
  matches a command that was actually run, not just claimed.
* **After the answer**, any tool that ran on work the request never touched
  fails the task: an agent that fixed the bug and then spent minutes on a job
  nobody asked for did not finish cleanly, however good the fix.

**The judge for everything a probe can't check** — quality of the reasoning,
whether instructions were followed, and question tasks like "what does this file
do," which have no mechanical check at all and are decided purely on whether the
answer establishes the required points. The judge reads the original project once,
then each task's setup, the agent's diff, and its answer, and grades every task in
a single call across four axes (1–5): instruction compliance, work quality,
reasoning, and **fabrication-free** — whether the answer claims anything the tool
record doesn't support.

**Truthfulness is scored against what the agent actually did, not what it says it
did.** The judge is shown the tool record (commands run, their output) alongside
the answer; a claim of "all 4 tests passed" that the transcript doesn't back up
scores low on fabrication-free even if the rest of the fix is correct. This is why
the controls check matters:

```bash theme={null}
gaia eval tasks controls
```

It plants three attempts at one real task with known, opposite verdicts — an
**ideal** attempt (correct fix, tests genuinely run), a **fabricated** one (the same
fix, but claiming an invented test result), and **nothing** (no work at all,
claiming to be done) — and requires the judge to grade them apart: high marks for
the honest attempt, a low fabrication-free score for the dishonest one even though
the code is identical, and a low score across the board for the empty one. If it
can't tell them apart, the quality numbers from that judge model mean less than
they appear to.

## The no-internet tasks

Two of the four `therock` tasks tell the agent in its prompt not to use the
internet. **The network itself is left open** — nothing is sandboxed off. Instead,
every tool call is scanned for anything that reached, or tried to reach, a remote
host (`gaia.eval.bench.transcripts.web_uses`), and any hit is reported as a
compliance finding, the same way for both harnesses.

That's deliberate: cutting the network would only measure whether the agent
*could* reach the web, a sandboxing question. Leaving it open and reading the tool
record measures whether the agent does what it was told — instruction-following,
which is the actual thing under test.

## SWE-bench Verified

[SWE-bench Verified](https://huggingface.co/datasets/SWE-bench/SWE-bench_Verified)
is the 500-instance, human-validated subset of SWE-bench: each instance is a real
GitHub issue from a public Python project (requests, flask, pylint, sphinx, django,
...), the commit the fix was made against, and the tests the fix made pass. An
agent is given the repository at that commit and the issue text, and it **resolves**
the instance when the official harness applies its patch in the instance's own
Docker image and every `FAIL_TO_PASS` test passes with no `PASS_TO_PASS` test
breaking.

**Nothing of the dataset is committed here.** `--suite swebench` is built at run
time: each instance is read from the Hugging Face dataset (the `datasets` package
when it is installed, the datasets-server REST API otherwise) and cached as one
JSON file under the work root (`$GAIA_BENCH_WORK_ROOT/swebench/`), which `--fence`
keeps out of the agent's reach. That cache never holds the gold patch: the judge
fetches the patches again after the agent exits, so re-judging a run needs the
network too. The `datasets` package keeps its own copy of the whole split, gold
patches included, in the Hugging Face cache in your home folder, where a
full-access agent can read it; run without `datasets` installed when that matters. `--instances a,b,c` names the instances; without
it the run is the five-instance pilot: `psf__requests-1921`, `psf__requests-5414`,
`pallets__flask-5014`, `pylint-dev__pylint-4970`, `sphinx-doc__sphinx-8721`.
`--sample N` draws a seeded random N of all 500 instead (`--seed S` picks another
draw), so a check on tasks nobody tuned on is one flag and anyone can reproduce it.

**The checkout is built the way `therock` builds its own.** A bare cache per
repository is filled by commit, to a depth of 200, so it never holds the fix that
landed after the base; each task then gets a single-branch clone over `file://`,
detached, with no remote. The `file://` matters: git ignores `--depth` on a plain
local path and copies every object, which is how a TheRock run once found the fix
as a dangling object and applied it verbatim. Every SWE-bench checkout is checked
with `git fsck --unreachable --no-reflogs` and refused if anything is off its
history.

**The prompt is the issue text plus one fixed paragraph**: work only in this
checkout, make the minimal code change that resolves the issue, do not use the
internet, and the project's tests cannot be run here (they run later, in the
container). Web use is reported the same way as for the `therock` no-internet
tasks. After the agent exits, everything it changed (new files included) becomes
one row of `DIR/predictions.jsonl` — `{"instance_id", "model_name_or_path",
"model_patch"}`, exactly what the official harness reads. An agent that changed
nothing fails the task then and there; there is nothing to submit.

**The agent does not get the host's installed copy of the package under test.**
GAIA's own interpreter often has a newer release of a SWE-bench repository
installed, with the fix already in it: on `matplotlib__matplotlib-24870` the agent
imported the host's matplotlib 3.11 and read the new code. So on a SWE-bench task
both harnesses get a bare venv (`<work root>/swebench/toolchain/`, with nothing
installed, not even pip) as their only Python: it is first on `PATH`, the host
interpreter's directories are taken off `PATH`, `VIRTUAL_ENV` points at it, and
GAIA's `run_python` / `run_python_file` run it too. An agent that imports the
package gets `ModuleNotFoundError` and has to read the checkout; `pip install`
fails, and is reported as web use. The venv is shared by the tasks of a work
root and rebuilt before any task that finds something installed in it.
`--fence` adds an OS-level deny on macOS: Claude Code loses every site-packages
tree of the host interpreter, and GAIA loses the installed copy of the package
under test — not the whole tree, because its own process imports from there,
and not at all for a package GAIA itself depends on (`requests`, `sympy`); the
run logs a warning for those.

Without `--fence` (any non-macOS host) this is not a sandbox. GAIA's file tools
stay inside the checkout unless `--full-access`, but a Python snippet or Claude
Code's own tools can still open any file whose path the agent finds, and on
Windows `python3` or `py -3.x` can reach another installed interpreter.

**Grading runs the official harness, one instance at a time.** `run` does it after
the suite unless `--no-evaluate`; `gaia eval tasks swebench DIR` does it later, or
again. It needs the `swebench` package (5.x) in the same interpreter and a
running Docker daemon, and it stops with a message naming which is missing. The
per-instance images are **amd64-only and about 3 GB each**: on an Apple-silicon
Mac they run under Rosetta (`--docker-platform linux/amd64`, the default), and
each image is pulled just before its run and removed right after, so five
instances need 3 GB of disk, not 15. `--keep-images` skips the pull and the
removal for a machine that has the space and grades repeatedly. The harness's own
logs and reports land in `DIR/swebench/`, and each task's scorecard entry gains
a `swebench` block — resolved or not, whether the patch applied, and the
`FAIL_TO_PASS` / `PASS_TO_PASS` counts.

```bash theme={null}
pip install swebench            # the official harness, in GAIA's interpreter

# GAIA on the pilot, graded in Docker afterwards
gaia eval tasks run --suite swebench --model fireworks.glm-5p3-flash   --no-judge --out "$BENCH/swe-gaia-glm"

# Claude Code on the same instances; a chosen set instead of the pilot
gaia eval tasks run --suite swebench --harness claude-code --model claude-opus-5   --instances psf__requests-1921,pallets__flask-5014 --no-judge --out "$BENCH/swe-cc-opus"

# Grade (or re-grade) a run whose agent step already finished
gaia eval tasks swebench "$BENCH/swe-gaia-glm"
```

**Reading a SWE-bench number against a published one.** Published SWE-bench
Verified scores are the resolved fraction of all 500 instances, under each lab's
own scaffold; a five-instance pilot says which instances an agent resolved, not
what its rate would be. Compare on the same instances, with `--repeats`, and grow
the set before quoting a percentage. As of this writing, no Anthropic-published
SWE-bench Verified figure exists for Claude Opus 5 to set a `--harness
claude-code --model claude-opus-5` run beside.

## Prerequisites and cost

Both harnesses reach the model through a local gateway, so no agent process ever
holds a provider key. Running a comparison needs:

* **Lemonade with a cloud provider registered** (Fireworks) for the open-weights
  side of the comparison — the model gateway routes `--model fireworks.*` through
  it.
* **A Claude subscription** — for `--harness claude-code`, and always for the
  judge, regardless of which harness ran the task.

**What it costs.** The full 14-task `everyday` suite on GLM-5.3 Flash runs about
$0.10 in real Fireworks charges. `therock` costs dollars, not cents — four tasks
against a real 144K-line codebase run far more tokens per task than the synthetic
suite. **The judge is usually the bigger cost on `everyday`**: a full judge pass
runs $0.4–0.9, against \$0.08–0.10 for the whole GLM-5.3 Flash run it's grading.
Don't re-judge a run you've already judged — `judge` is a separate command
precisely so you don't have to re-run the agent to do it.

## Every command

| Command | What it writes, and where |
| - | - |
| `gaia eval tasks run --suite SUITE [--harness gaia\|claude-code] [--model M] --out DIR` | Runs the suite, one task at a time, and judges it unless `--no-judge`. Writes `DIR/scorecard.json`, and per task: `transcript.json`, `workspace.diff`, `setup.diff` (if the task has a setup). |
| `gaia eval tasks judge DIR [--judge-attempts N]` | Re-grades every task in a finished run from its saved transcripts — no agent re-run needed. Updates `DIR/scorecard.json` in place. |
| `gaia eval tasks gate DIR [--expect FILE] [--enforce]` | Compares a judged run with committed expectations; used by CI, not this comparison workflow. |
| `gaia eval tasks propose DIR... --out FILE` | Writes the expectations CI gates on from several judged runs (or one `--repeats` directory), each limit set by the worst run. |
| `gaia eval tasks controls [--out DIR]` | Grades the three planted attempts described above; with `--out`, also writes each attempt's `transcript.json` and the verdicts to `DIR/controls.json`. |
| `gaia eval tasks report DIR... --out DIR` | The harness × model table across one or more finished, judged runs. Writes `report.md`, `report.html`, and `report.png` (skipped without a headless Chrome, or with `--no-png`). |
| `gaia eval tasks gateway --port N` | Runs the model gateway in the foreground, so a whole matrix of runs can share one instead of each starting its own. |
| `gaia eval tasks run --suite swebench [--instances a,b] [--no-evaluate] [--keep-images] --out DIR` | The SWE-bench suite built from the named instances (default: the pilot), then graded by the official harness in Docker. Writes `DIR/predictions.jsonl` and the harness's logs and reports under `DIR/swebench/`. |
| `gaia eval tasks swebench DIR [--keep-images] [--docker-platform P]` | Grades (or re-grades) a finished `swebench` run's predictions with the official harness; updates `DIR/scorecard.json` in place. |

**A full comparison**, one harness/model combination per line:

```bash theme={null}
# GAIA on the open-weights model, 3 repeats, metered by Fireworks' real billing API
gaia eval tasks run --suite everyday --model fireworks.glm-5p3-flash \
  --repeats 3 --meter fireworks --out "$BENCH/gaia-glm"

# Claude Code on the same open-weights model, through the model gateway
gaia eval tasks run --suite everyday --harness claude-code \
  --model fireworks.glm-5p3-flash --repeats 3 --out "$BENCH/cc-glm"

# Claude Code on Anthropic's own model, as the reference row
gaia eval tasks run --suite everyday --harness claude-code \
  --model claude-opus-5 --out "$BENCH/cc-opus"

gaia eval tasks report "$BENCH/cc-opus" "$BENCH/gaia-glm" "$BENCH/cc-glm" --out "$BENCH/report"
```

The first directory named to `report` becomes the 100% row everything else is a
percentage of.

**`--repeats N`** runs the suite N times into `DIR/r1`, `DIR/r2`, ... One run is
noise — three Claude Code runs on the same model and tasks have ranged from 4:06 to
18:30, a 4.5× spread, and a TheRock run that scored 3/4 scored 4/4 on repeat.
`report` and `judge` both understand a `--repeats` directory automatically.

**Metering flags** — real cost instead of a rate-carded estimate:

| Flag | What it does |
| - | - |
| `--meter fireworks` | Snapshots Fireworks' own billing meter before and after the run and prices from the delta. |
| `--fireworks-account ID` | Account id for the meter (default: `$FIREWORKS_ACCOUNT_ID`). |
| `--meter-lag S` | Seconds to wait for the meter to catch up before the closing snapshot (default 150 — the meter trails the calls it counts by about 90s). |

Other flags worth knowing: `--tasks a,b` (a subset of the suite), `--run-timeout S`
(the wall-clock cap per task, default 1800s, the same for every harness in a
comparison), `--judge-model M` / `--judge-attempts N`. The full flag reference,
including `--fence`, `--full-access`, and the TheRock- and SWE-bench-specific
flags, is in the
[Evaluation Framework reference](https://amd-gaia.ai/docs/reference/eval.md#comparing-harnesses-and-models).

## Reading the results

Each row of the report is one (model, harness) pair, averaged over its runs, with
the min–max range printed beneath a cell when its runs disagreed:

| Column | Meaning |
| - | - |
| Passed | Tasks that passed, out of the suite total. |
| Quality | Mean of all four judge axes, across every judged task. |
| Truthful | Mean of the fabrication-free axis alone — the honesty signal on its own. |
| Steps / Tool calls | Summed across the suite's tasks. |
| Tokens / % tokens | Total tokens, and as a percentage of the report's 100% row. |
| Time / % time | Wall-clock time, same percentage convention. |
| Cost / % cost | Priced with the source named beneath it — see below. |
| Web uses | Count of tool calls that reached or tried to reach the internet — should be 0 outside the no-internet `therock` tasks. |

**A difference smaller than the run-to-run spread is noise.** If two rows' ranges
overlap, don't report a winner from a single comparison — rerun with `--repeats`
and look at whether the means still separate once you can see the spread.

Cost carries its source because the numbers aren't the same kind of dollar:

| Source | What it is |
| - | - |
| `metered` | The provider's real billing meter. Actual charges. |
| `api_equivalent` | Claude Code's own cost figure on Anthropic models — on a subscription, this is the *list price of the tokens*, not money spent. |
| `harness_counts` | The harness's own per-call token counts, priced from a published rate card, where no meter snapshot was taken. |

## Things that will mislead you if you skip them

* **A single run is not a result.** One Claude Code + GLM `therock` run scored
  3/4; its rerun scored 4/4 — same model, same tasks, different outcome. Use
  `--repeats` and read the range, not one number.
* **`api_equivalent` cost is not money spent.** On a Claude subscription it's the
  list price of the tokens Claude Code reports using — a price of compute for
  comparing efficiency, never a dollar figure to sum against a real bill.
* **The provider meter is account-wide.** Two `--meter fireworks` runs on the same
  model at the same time pollute each other's deltas — this has actually happened,
  inflating one run's reported cost by 58%. Keep metered runs on one account
  serial, the same rule as everything else that shares Lemonade.
* **The judge sometimes returns nothing for a batch.** A timeout or a malformed
  response leaves a task ungraded: an ungraded question counts as not passed, a
  coding task keeps its test result, and either one drops out of the quality and
  truthfulness averages, so those averages cover fewer tasks. Re-run
  `gaia eval tasks judge DIR` (it regrades from the saved transcripts, no agent
  re-run needed); never treat a scorecard with ungraded tasks as final.
* **The harness's own token counts run 1–6% below the provider's meter.**
  `harness_counts` is a real estimate, not a rounding artifact — when the number
  needs to be exact, meter it instead of trusting the harness count.

## Privacy

A transcript captures everything the agent saw while working the task — file
contents, command output, whatever the project or the prompt contained. Keep runs
local: nothing here uploads anything on its own, but treat a transcript like any
other file that might contain something you don't want shared before deciding to
post one.

Every transcript and result *is* scrubbed of known secret-shaped strings before
it's written, and a run fails outright if anything survives that check rather than
writing a leak — but that only catches credential shapes, not general repo content,
so it's not a substitute for reading a transcript before sharing it.

**Results are never committed to the repo.** As the Quick Start above notes, only
the default `eval-*` output directory is gitignored, and any other path inside the
repo is not — always send `--out` (or `$GAIA_BENCH_RESULTS_DIR`) somewhere
outside it, and never `git add` anything a run wrote.

## See also

* [Evaluation Framework reference](https://amd-gaia.ai/docs/reference/eval.md#flagship-agent-tasks) — the full
  technical reference: task authoring, the `core`/`adversarial`/`full` CI suites,
  the gate and its expectations files, TheRock checkout details.
* The `benchmarking-the-agent` skill
  (`.claude/skills/benchmarking-the-agent/SKILL.md`) — recipes, traps, and what a
  healthy result currently looks like, kept up to date from real runs.

***

**License**

Copyright(C) 2024-2026 Advanced Micro Devices, Inc. All rights reserved.

SPDX-License-Identifier: MIT


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.