Testing

Overview

The C++ framework has two test suites:

Suite	Binary	LLM Required	Hardware	What it tests
Unit tests	`tests_mock`	No	Any	All core modules with mocked dependencies
Integration tests	`tests_integration`	Yes	Windows + LLM server	Real LLM calls, MCP connections, agent workflows

Both suites use Google Test (fetched automatically by CMake).

Unit Tests

Unit tests are built by default and run without an LLM server. They cover all twenty core modules:

Module	Test file	What it covers
Agent loop	`test_agent.cpp`	State machine, multi-step planning, error recovery
Agent (VLM)	`test_agent_vlm.cpp`	`processQuery` image overloads, history stripping
Tool registry	`test_tool_registry.cpp`	Registration, lookup, execution, parameter schemas
Tool integration	`test_tool_integration.cpp`	End-to-end tool calling with mocked LLM
JSON utilities	`test_json_utils.cpp`	Multi-strategy JSON extraction, fallback parsing
JSON event handler	`test_json_event_handler.cpp`	Structured JSONL event emission (thought/goal/answer)
MCP client	`test_mcp_client.cpp`	JSON-RPC protocol, stdio transport, tool discovery
Console	`test_console.cpp`	TerminalConsole and SilentConsole output
Clean console	`test_clean_console.cpp`	CleanConsole TUI, word-wrap, ANSI colors
Types	`test_types.cpp`	AgentConfig, Message, ToolInfo, ParsedResponse
Image	`test_image.cpp`	Image loading, MIME detection, base64 encoding
Decision types	`test_decision.cpp`	`Decision` struct and interactive menu parsing
Security	`test_security.cpp`	`ToolPolicy`, `validatePath`, shell-arg sanitisation, `AllowedToolsStore`
SSE parser	`test_sse_parser.cpp`	OpenAI-style streaming-event parsing and recovery
Lemonade client	`test_lemonade_client.cpp`	Lemonade HTTP surface (health, models, chat completions)
Process runner	`test_process.cpp`	Cross-platform subprocess execution and pipes
File tools	`test_file_tools.cpp`	`file_read`/`file_write`/`file_edit`/`file_search`
Git tools	`test_git_tools.cpp`	`git_status`/`git_diff`/`git_log`/`git_show`
Session store	`test_session.cpp`	Session persistence (save/list/resume)
REPL	`test_repl.cpp`	Interactive REPL loop and slash-command parsing

One additional module, test_tui_console.cpp (FTXUI TUI console), is compiled into tests_mock only when GAIA_BUILD_TUI=ON (the default).

Building Unit Tests

Unit tests are built by default when building the C++ framework as a standalone project (GAIA_BUILD_TESTS=ON). The standard build commands from the Quickstart produce the tests_mock binary automatically:

Windows (MSVC)
Linux

cd cpp

cmake -B build -G "Visual Studio 17 2022" -A x64

cmake --build build --config Release

Binary: build\Release\tests_mock.exe

cd cpp

cmake -B build -DCMAKE_BUILD_TYPE=Release

cmake --build build

Binary: build/tests_mock

If you’re consuming gaia_core as a sub-project (via FetchContent or add_subdirectory), tests are off by default. Pass -DGAIA_BUILD_TESTS=ON to enable them.

Running Unit Tests

Windows
Linux

Run the test binary directly:

build\Release\tests_mock.exe --gtest_color=yes

Or via CTest:

ctest --test-dir build -C Release --output-on-failure

Run the test binary directly:

./build/tests_mock --gtest_color=yes

Or via CTest:

ctest --test-dir build --output-on-failure

Filtering Tests

Google Test supports test filtering via --gtest_filter:

# Run only agent tests
tests_mock --gtest_filter="AgentTest.*"

# Run only JSON utility tests
tests_mock --gtest_filter="JsonUtils.*"

# Run a specific test
tests_mock --gtest_filter="ToolRegistryTest.RegisterAndExecute"

Integration Tests

Integration tests exercise the full stack — real LLM inference, live MCP server connections, and actual agent workflows. They require:

A running LLM server (Lemonade recommended) with a model loaded
Windows (the tested agents use PowerShell)
uvx on PATH (for MCP tests)

Test file	What it covers
`test_integration_llm.cpp`	LLM chat completions, tool-calling responses
`test_integration_mcp.cpp`	MCP server connection, tool discovery, tool execution
`test_integration_wifi.cpp`	Wi-Fi agent end-to-end diagnostic flow
`test_integration_health.cpp`	Health agent MCP-based system diagnostics
`test_integration_vlm.cpp`	Vision-language inference against a real VLM model

Building Integration Tests

Integration tests are not built by default. Enable them with the GAIA_BUILD_INTEGRATION_TESTS CMake option:

cmake -B build -G "Visual Studio 17 2022" -A x64 -DGAIA_BUILD_INTEGRATION_TESTS=ON

cmake --build build --config Release

This produces the tests_integration binary alongside the unit test binary.

Do not mix unit and integration tests in CTest. If you enable integration tests, CTest will discover both suites. Run tests_integration separately with an LLM server running — otherwise CTest will hang waiting for the LLM connection.

Running Integration Tests

Start an LLM server with a model loaded:
lemonade-server serve

Set environment variables (optional — defaults shown):

set GAIA_CPP_BASE_URL=http://localhost:13305/api/v1

set GAIA_CPP_TEST_MODEL=Qwen3-4B-Instruct-2507-GGUF

Run the tests:

build\Release\tests_integration.exe --gtest_color=yes

Integration tests have a 300-second timeout per test. If a test hangs, verify the LLM server is responding at the configured URL.

CI Pipeline

The C++ CI workflow (.github/workflows/build_cpp.yml) runs five jobs on every PR that touches cpp/:

Job	Runs on	What it validates
Build & Test	Ubuntu + Windows (cloud)	CMake build + unit tests
Install Test	Ubuntu + Windows (cloud)	`cmake --install` + `find_package` round-trip
Shared Library	Ubuntu + Windows (cloud)	`BUILD_SHARED_LIBS=ON` compiles cleanly
Integration Tests	STX hardware (self-hosted)	Full stack: LLM + MCP + WiFi + Health
Benchmarks	Ubuntu + Windows (cloud)	Performance benchmarks (`benchmark_cpp.yml`), runs after Build & Test succeeds

The integration test job runs on AMD STX hardware with a real Lemonade Server and Qwen3-4B model. It is treated as non-blocking in CI (infrastructure issues on self-hosted runners don’t fail the PR).

Adding New Tests

Unit Tests

Add your test file to cpp/tests/ and register it in CMakeLists.txt:

add_executable(tests_mock
    # ... existing test files ...
    tests/test_your_module.cpp   # Add here
)

Follow the existing pattern — include <gtest/gtest.h>, use TEST() or TEST_F() macros, and mock LLM responses where needed. See test_agent.cpp for mocking examples.

Integration Tests

Add your test file to cpp/tests/integration/ and register it in the tests_integration target in CMakeLists.txt:

add_executable(tests_integration
    # ... existing test files ...
    tests/integration/test_integration_your_feature.cpp   # Add here
)

Integration tests should:

Read GAIA_CPP_BASE_URL and GAIA_CPP_TEST_MODEL from environment variables
Use reasonable timeouts (the CI sets 300s per test)
Be idempotent — no permanent side effects

Getting Started

C++ Framework

Python Framework

Overview

Unit Tests

Building Unit Tests

Running Unit Tests

Filtering Tests

Integration Tests

Building Integration Tests

Running Integration Tests

CI Pipeline

Adding New Tests

Unit Tests

Integration Tests

​Overview

​Unit Tests

​Building Unit Tests

​Running Unit Tests

​Filtering Tests

​Integration Tests

​Building Integration Tests

​Running Integration Tests

​CI Pipeline

​Adding New Tests

​Unit Tests

​Integration Tests

Overview

Unit Tests

Building Unit Tests

Running Unit Tests

Filtering Tests

Integration Tests

Building Integration Tests

Running Integration Tests

CI Pipeline

Adding New Tests

Unit Tests

Integration Tests