Skip to main content

Testing

17.1 Testing Agents


17.2 Silent Mode for Testing


17.3 Mocking LLM Responses

Consider using the built-in MockLLMProvider / MockVLMClient helpers in gaia.testing (src/gaia/testing/mocks.py) for richer fixtures. The require_lemonade pytest fixture in tests/conftest.py skips integration tests automatically if no Lemonade server is running.

Behavior-E2E — Assert side-effects, not replies

The class of bug “agent claims success but the tool never ran” (see #1428) is invisible to:
  • Unit tests that mock the LLM and feed clean tool calls
  • UI render E2E tests that assert “the agent replied”
The pattern to catch it: drive each tool-using agent through the real server with a real model, repeat each scenario N× (the failure is output-format-dependent and non-deterministic), and assert the tool’s side-effect actually occurred — not merely that the agent produced a reply.
  • False-success is a hard fail: a “success” reply with no side-effect is worse than honest failure.
  • Planted unguessable facts: use secrets.token_hex(4) in the prompt so a cached or hallucinated reply cannot accidentally pass.
  • N× repetition: repeat ≥5 times; treat any single run with a missing side-effect as a failure.