Skip to main content
Source Code: src/gaia/eval/bench/, src/gaia/eval/flagship_tasks.py, eval/tasks/tasks.json

​
What this is

gaia eval tasks gives an autonomous coding agent real work — fix a bug, triage issues, write a weekly report from a CSV — then checks what actually happened: did the work get done, was the agent honest about it, and what did it cost. The same fixed set of tasks, time limit, tools and judge run against two harnesses — the flagship GaiaAgent, or Claude Code driven headlessly — so a difference in the numbers is a difference between agents, not between conditions. This is the tool behind claims like “GLM-5.3 Flash under GAIA scores within about a tenth of a point of Claude Opus 5 under Claude Code, for a fraction of the cost.” It answers them with a reproducible run anyone can repeat, not a one-off transcript nobody else can check.
This drives a real agent against real models — it spends real tokens and, on a metered provider, real money. It is a contributor/maintainer tool for comparing harnesses and models, not something an end user runs.

​
Prerequisites

  • The flagship agent installed: pip install -e hub/agents/gaia/python (see the Dev Guide if you haven’t set up the base install).
  • A running Lemonade Server with a cloud provider registered — Fireworks, for the open-weights models these benchmarks target. gaia eval tasks reaches every model through Lemonade’s OpenAI-compatible endpoint.
  • The Claude Code CLI (claude) on PATH, signed in to a subscription. It does two jobs here, both riding the subscription rather than an API key: it drives the --harness claude-code leg of a comparison, and it is always the judge, no matter which harness ran the task.
Nothing here needs ANTHROPIC_API_KEY — if claude already works on its own, you’re set.

​
Quick start

The default output directory, eval/results/eval-tasks-<timestamp>, is gitignored, but only by its eval-* name: an --out elsewhere in the repo is not. Point --out (or $GAIA_BENCH_RESULTS_DIR) outside the repo, as below, and never commit anything a run writes.
1

Check the judge, before trusting any quality number

Plants three attempts at one real task — a correct fix with its test suite actually run, the same fix with an invented test run, and no work at all claiming to be done — and requires the judge to grade them apart. Takes under a minute.
2

Run one task under GAIA's own harness

--tasks narrows a suite to specific task ids, so a first try doesn’t spend the full 14-task run. This one needs the judge (it’s a question task with no mechanical check), so don’t add --no-judge.
Writes scorecard.json, plus 21-qa/transcript.json and 21-qa/workspace.diff, under $BENCH/first-run.
3

Render the report

Builds the harness × model table from one or more finished runs. With a single run directory it’s a one-row table — useful to confirm the pipeline end to end before running a real comparison.
Writes report.md, report.html, and report.png where a headless Chrome is on the machine.

​
Harness vs. model

A model is the LLM doing the thinking — GLM-5.3 Flash, Claude Opus 5, Gemma. A harness is the agent loop wrapped around it: how it reads the task, decides which tool to call, and decides when it’s done. GAIA has its own harness (the flagship GaiaAgent); Claude Code is a different one. The same model can score differently under each, because the harness — not just the model — decides how well that thinking turns into finished work. --harness gaia|claude-code is what isolates that: same tasks, same time limit, same judge, same toolchain on PATH, only the harness changes. Running the same model through both legs is how you find out whether a weak score is the model or the wrapper around it.

​
The three benchmarks

everyday is the one to reach for first — small, fast, cheap, and mechanically checkable for most tasks. therock is a real 144K-line codebase with a real fix as the answer key; it costs more and takes longer, and it’s what proves the agent’s skills transfer past a project built for the benchmark. swebench is the public yardstick: the same instances, prompt discipline and grader everyone else reports against, so a number from it can be set beside a published one.

​
How scoring works

Every everyday task starts from a fresh copy of toybox; the agent may read and write that copy and nothing else. Scoring runs in two stages: Mechanical checks first, for tasks where the project itself can decide:
  • tests_pass — the project’s own test suite passes.
  • probe — a short snippet run inside the finished project, checking the actual behavior (e.g. the bug fix now accepts a lowercase input the original rejected).
  • unchanged — named files are byte-identical to the pre-run snapshot, for tasks where touching them is itself a failure (a code-review task that edits the file under review, say).
  • Verified additionally checks the agent’s own tool record for a pytest run that passed after its last file edit — proof the “4 tests pass” in the answer matches a command that was actually run, not just claimed.
  • After the answer, any tool that ran on work the request never touched fails the task: an agent that fixed the bug and then spent minutes on a job nobody asked for did not finish cleanly, however good the fix.
The judge for everything a probe can’t check — quality of the reasoning, whether instructions were followed, and question tasks like “what does this file do,” which have no mechanical check at all and are decided purely on whether the answer establishes the required points. The judge reads the original project once, then each task’s setup, the agent’s diff, and its answer, and grades every task in a single call across four axes (1–5): instruction compliance, work quality, reasoning, and fabrication-free — whether the answer claims anything the tool record doesn’t support. Truthfulness is scored against what the agent actually did, not what it says it did. The judge is shown the tool record (commands run, their output) alongside the answer; a claim of “all 4 tests passed” that the transcript doesn’t back up scores low on fabrication-free even if the rest of the fix is correct. This is why the controls check matters:
It plants three attempts at one real task with known, opposite verdicts — an ideal attempt (correct fix, tests genuinely run), a fabricated one (the same fix, but claiming an invented test result), and nothing (no work at all, claiming to be done) — and requires the judge to grade them apart: high marks for the honest attempt, a low fabrication-free score for the dishonest one even though the code is identical, and a low score across the board for the empty one. If it can’t tell them apart, the quality numbers from that judge model mean less than they appear to.

​
The no-internet tasks

Two of the four therock tasks tell the agent in its prompt not to use the internet. The network itself is left open — nothing is sandboxed off. Instead, every tool call is scanned for anything that reached, or tried to reach, a remote host (gaia.eval.bench.transcripts.web_uses), and any hit is reported as a compliance finding, the same way for both harnesses. That’s deliberate: cutting the network would only measure whether the agent could reach the web, a sandboxing question. Leaving it open and reading the tool record measures whether the agent does what it was told — instruction-following, which is the actual thing under test.

​
SWE-bench Verified

SWE-bench Verified is the 500-instance, human-validated subset of SWE-bench: each instance is a real GitHub issue from a public Python project (requests, flask, pylint, sphinx, django, …), the commit the fix was made against, and the tests the fix made pass. An agent is given the repository at that commit and the issue text, and it resolves the instance when the official harness applies its patch in the instance’s own Docker image and every FAIL_TO_PASS test passes with no PASS_TO_PASS test breaking. Nothing of the dataset is committed here. --suite swebench is built at run time: each instance is read from the Hugging Face dataset (the datasets package when it is installed, the datasets-server REST API otherwise) and cached as one JSON file under the work root ($GAIA_BENCH_WORK_ROOT/swebench/), which --fence keeps out of the agent’s reach. That cache never holds the gold patch: the judge fetches the patches again after the agent exits, so re-judging a run needs the network too. The datasets package keeps its own copy of the whole split, gold patches included, in the Hugging Face cache in your home folder, where a full-access agent can read it; run without datasets installed when that matters. --instances a,b,c names the instances; without it the run is the five-instance pilot: psf__requests-1921, psf__requests-5414, pallets__flask-5014, pylint-dev__pylint-4970, sphinx-doc__sphinx-8721. --sample N draws a seeded random N of all 500 instead (--seed S picks another draw), so a check on tasks nobody tuned on is one flag and anyone can reproduce it. The checkout is built the way therock builds its own. A bare cache per repository is filled by commit, to a depth of 200, so it never holds the fix that landed after the base; each task then gets a single-branch clone over file://, detached, with no remote. The file:// matters: git ignores --depth on a plain local path and copies every object, which is how a TheRock run once found the fix as a dangling object and applied it verbatim. Every SWE-bench checkout is checked with git fsck --unreachable --no-reflogs and refused if anything is off its history. The prompt is the issue text plus one fixed paragraph: work only in this checkout, make the minimal code change that resolves the issue, do not use the internet, and the project’s tests cannot be run here (they run later, in the container). Web use is reported the same way as for the therock no-internet tasks. After the agent exits, everything it changed (new files included) becomes one row of DIR/predictions.jsonl — {"instance_id", "model_name_or_path", "model_patch"}, exactly what the official harness reads. An agent that changed nothing fails the task then and there; there is nothing to submit. The agent does not get the host’s installed copy of the package under test. GAIA’s own interpreter often has a newer release of a SWE-bench repository installed, with the fix already in it: on matplotlib__matplotlib-24870 the agent imported the host’s matplotlib 3.11 and read the new code. So on a SWE-bench task both harnesses get a bare venv (<work root>/swebench/toolchain/, with nothing installed, not even pip) as their only Python: it is first on PATH, the host interpreter’s directories are taken off PATH, VIRTUAL_ENV points at it, and GAIA’s run_python / run_python_file run it too. An agent that imports the package gets ModuleNotFoundError and has to read the checkout; pip install fails, and is reported as web use. The venv is shared by the tasks of a work root and rebuilt before any task that finds something installed in it. --fence adds an OS-level deny on macOS: Claude Code loses every site-packages tree of the host interpreter, and GAIA loses the installed copy of the package under test — not the whole tree, because its own process imports from there, and not at all for a package GAIA itself depends on (requests, sympy); the run logs a warning for those. Without --fence (any non-macOS host) this is not a sandbox. GAIA’s file tools stay inside the checkout unless --full-access, but a Python snippet or Claude Code’s own tools can still open any file whose path the agent finds, and on Windows python3 or py -3.x can reach another installed interpreter. Grading runs the official harness, one instance at a time. run does it after the suite unless --no-evaluate; gaia eval tasks swebench DIR does it later, or again. It needs the swebench package (5.x) in the same interpreter and a running Docker daemon, and it stops with a message naming which is missing. The per-instance images are amd64-only and about 3 GB each: on an Apple-silicon Mac they run under Rosetta (--docker-platform linux/amd64, the default), and each image is pulled just before its run and removed right after, so five instances need 3 GB of disk, not 15. --keep-images skips the pull and the removal for a machine that has the space and grades repeatedly. The harness’s own logs and reports land in DIR/swebench/, and each task’s scorecard entry gains a swebench block — resolved or not, whether the patch applied, and the FAIL_TO_PASS / PASS_TO_PASS counts.
Reading a SWE-bench number against a published one. Published SWE-bench Verified scores are the resolved fraction of all 500 instances, under each lab’s own scaffold; a five-instance pilot says which instances an agent resolved, not what its rate would be. Compare on the same instances, with --repeats, and grow the set before quoting a percentage. As of this writing, no Anthropic-published SWE-bench Verified figure exists for Claude Opus 5 to set a --harness claude-code --model claude-opus-5 run beside.

​
Prerequisites and cost

Both harnesses reach the model through a local gateway, so no agent process ever holds a provider key. Running a comparison needs:
  • Lemonade with a cloud provider registered (Fireworks) for the open-weights side of the comparison — the model gateway routes --model fireworks.* through it.
  • A Claude subscription — for --harness claude-code, and always for the judge, regardless of which harness ran the task.
What it costs. The full 14-task everyday suite on GLM-5.3 Flash runs about $0.10 in real Fireworks charges. `therock` costs dollars, not cents — four tasks against a real 144K-line codebase run far more tokens per task than the synthetic suite. **The judge is usually the bigger cost on `everyday`**: a full judge pass runs $0.4–0.9, against $0.08–0.10 for the whole GLM-5.3 Flash run it’s grading. Don’t re-judge a run you’ve already judged — judge is a separate command precisely so you don’t have to re-run the agent to do it.

​
Every command

A full comparison, one harness/model combination per line:
The first directory named to report becomes the 100% row everything else is a percentage of. --repeats N runs the suite N times into DIR/r1, DIR/r2, … One run is noise — three Claude Code runs on the same model and tasks have ranged from 4:06 to 18:30, a 4.5× spread, and a TheRock run that scored 3/4 scored 4/4 on repeat. report and judge both understand a --repeats directory automatically. Metering flags — real cost instead of a rate-carded estimate: Other flags worth knowing: --tasks a,b (a subset of the suite), --run-timeout S (the wall-clock cap per task, default 1800s, the same for every harness in a comparison), --judge-model M / --judge-attempts N. The full flag reference, including --fence, --full-access, and the TheRock- and SWE-bench-specific flags, is in the Evaluation Framework reference.

​
Reading the results

Each row of the report is one (model, harness) pair, averaged over its runs, with the min–max range printed beneath a cell when its runs disagreed: A difference smaller than the run-to-run spread is noise. If two rows’ ranges overlap, don’t report a winner from a single comparison — rerun with --repeats and look at whether the means still separate once you can see the spread. Cost carries its source because the numbers aren’t the same kind of dollar:

​
Things that will mislead you if you skip them

  • A single run is not a result. One Claude Code + GLM therock run scored 3/4; its rerun scored 4/4 — same model, same tasks, different outcome. Use --repeats and read the range, not one number.
  • api_equivalent cost is not money spent. On a Claude subscription it’s the list price of the tokens Claude Code reports using — a price of compute for comparing efficiency, never a dollar figure to sum against a real bill.
  • The provider meter is account-wide. Two --meter fireworks runs on the same model at the same time pollute each other’s deltas — this has actually happened, inflating one run’s reported cost by 58%. Keep metered runs on one account serial, the same rule as everything else that shares Lemonade.
  • The judge sometimes returns nothing for a batch. A timeout or a malformed response leaves a task ungraded: an ungraded question counts as not passed, a coding task keeps its test result, and either one drops out of the quality and truthfulness averages, so those averages cover fewer tasks. Re-run gaia eval tasks judge DIR (it regrades from the saved transcripts, no agent re-run needed); never treat a scorecard with ungraded tasks as final.
  • The harness’s own token counts run 1–6% below the provider’s meter. harness_counts is a real estimate, not a rounding artifact — when the number needs to be exact, meter it instead of trusting the harness count.

​
Privacy

A transcript captures everything the agent saw while working the task — file contents, command output, whatever the project or the prompt contained. Keep runs local: nothing here uploads anything on its own, but treat a transcript like any other file that might contain something you don’t want shared before deciding to post one. Every transcript and result is scrubbed of known secret-shaped strings before it’s written, and a run fails outright if anything survives that check rather than writing a leak — but that only catches credential shapes, not general repo content, so it’s not a substitute for reading a transcript before sharing it. Results are never committed to the repo. As the Quick Start above notes, only the default eval-* output directory is gitignored, and any other path inside the repo is not — always send --out (or $GAIA_BENCH_RESULTS_DIR) somewhere outside it, and never git add anything a run wrote.

​
See also

  • Evaluation Framework reference — the full technical reference: task authoring, the core/adversarial/full CI suites, the gate and its expectations files, TheRock checkout details.
  • The benchmarking-the-agent skill (.claude/skills/benchmarking-the-agent/SKILL.md) — recipes, traps, and what a healthy result currently looks like, kept up to date from real runs.