What this gives you
GAIA normally runs models Lemonade loads on your own machine. The AMD LLM gateway hosts models too large for a laptop — a Gemma-4-31B-class on-prem model, and frontier models like Claude — behind one OpenAI-compatible endpoint. Connecting it means every GAIA agent can use those models with no code change: they appear in Lemonade’s model list alongside your local ones, andgaia chat,
gaia llm, and the TUI address them the same way. That also makes it possible to run
the same task through a GAIA agent and through another tool on the same model, and
compare the results directly.
Where your token is kept
You enter it once. GAIA remembers it in your OS credential store — DPAPI on Windows, Keychain on macOS, SecretService on Linux — so it is encrypted at rest and scoped to your user account. GAIA refuses to fall back to a plaintext keyring backend, so it cannot quietly become a readable file. It is never written to a GAIA file.~/.gaia/gateway.json records the base URL,
which models you enabled, and what GAIA learned about them, and has no field a secret could occupy.
Lemonade itself holds the token in memory only and forgets it on restart. GAIA notices
and restores it from the credential store, or from LEMONADE_AMD_API_KEY /
GAIA_GATEWAY_TOKEN in GAIA’s environment when the gateway is AMD’s, which is why you
are not asked again.
gaia gateway install restores it too. Installing with a different --base-url deletes
the remembered token instead, so a token for one gateway is never sent to another; run
gaia gateway auth for the new one.
LEMONADE_AMD_API_KEY in Lemonade’s environment still works and takes precedence.
If it is set and you supply a different token, Lemonade refuses with a clear error
rather than silently picking one. Claude models read the token from GAIA, not Lemonade,
so set the same variable in GAIA’s environment too; see
Claude models and prompt caching.
Connect from the TUI
Type/gateway in chat (or pick it from the / command palette).
1
Confirm the gateway URL
The OpenAI-compatible base URL, e.g.
https://llm-api.amd.com/Unified/v1. GAIA tests it
before registering, so a wrong path fails immediately with the actual HTTP status
instead of showing up later as an empty model list.2
Paste your token
The field is masked. GAIA sends it to Lemonade and keeps no copy.
3
Done — a model is already selected
GAIA picks
DeepSeek-V4.1-Flash for you and makes it the default. GAIA’s
agent path streams, so the default has to be a model the gateway streams, and
this one does; it is also the fastest model on GAIA’s agent benchmark. If your
gateway doesn’t offer it, GAIA picks Gemma-4-31B, which streams too and is
on-prem, so it costs nothing per token.To use a different one: space enables a model, enter makes it
active. Preferred models are marked * and sorted first. A model whose last call
returned “not found” is marked (unavailable) and is never picked for you.Connect from the CLI
gaia gateway test names the model it is testing. If the gateway answers “not found”
(HTTP 404) for it, the test names the model and a few deployed ones to switch to with
gaia gateway use <id>.
gaia gateway status shows registration, auth state, discovered model count, your
current selection, and whether the active model’s last call failed.
When a model stops being served
Gateways retire and rename models. When a call to a gateway model returns HTTP 404, GAIA records it in~/.gaia/gateway.json with the time, and a later successful call
clears it. GAIA never switches to another model on its own; it tells you instead:
gaia chat,gaia llm, andgaia promptprint one warning line when your saved default model is the one that failed.gaia gateway statusandgaia gateway modelsmark it unavailable.- The TUI’s
/gatewayscreen marks it(unavailable), says so under the list when it is the active model, and never auto-selects it. - A chat turn in the Agent UI or the TUI ends with an error that names the model and
says retrying won’t help. The Agent UI points you to its model menu (or to
Settings > Advanced > Custom model ID if you typed the ID there); the TUI points
you to
/model.
Gateways with a different auth header
The default isOcp-Apim-Subscription-Key with no prefix, because AMD’s gateway
runs behind Azure API Management and that is the header it checks.
So a plain OpenAI-compatible endpoint expecting a bearer token is the case that needs
the override — not the other way round:
Using gateway models
Gateway models are namespacedamd.<model> and work anywhere a local model does:
Claude models and prompt caching
Chat with a Claude model (amd.Claude-*) skips Lemonade and goes straight to the
gateway’s Anthropic Messages route, https://llm-api.amd.com/Anthropic — the same route Claude
Code uses. The OpenAI-compatible route Lemonade calls ignores Anthropic’s cache markers,
so every agent step re-billed the whole prompt. On the Anthropic route, the tool
schemas, system prompt, and conversation so far are read from cache on later steps:
a repeated 48K-token prompt came back with all but 4 tokens cached.
gaia -p --output-format json reports this as cached_tokens.
What that means in practice:
- Same token as every other gateway model. GAIA sends it in the
Ocp-Apim-Subscription-Keyheader itself, readingLEMONADE_AMD_API_KEY, thenGAIA_GATEWAY_TOKENin GAIA’s environment, then the OS credential store thatgaia gateway authwrites to. When Lemonade forgets its token, GAIA gives it back this same one, so Claude and DeepSeek can’t end up on different keys. GAIA never sends an Anthropic API key there. - Lemonade gets the token in GAIA’s environment only for AMD’s gateway. If you
registered a different gateway, GAIA gives Lemonade only the token
gaia gateway authstored for it, and logs a warning that the environment token was left out. Installing with a different--base-urldeletes the stored token, so rungaia gateway authfor the new gateway. - Lemonade’s own
LEMONADE_AMD_API_KEYshould be in GAIA’s environment too. When Lemonade was started with that variable, it ignores tokens sent at runtime, and GAIA can’t read the key back. Claude models then send GAIA’s other copy (GAIA_GATEWAY_TOKENor a stored token) and log a warning that the two can drift. If that copy has gone stale, the 401 says so. SetLEMONADE_AMD_API_KEYto the same key in GAIA’s environment to make them one. gaia gateway authupdates both. It fails if Claude models would still send a different token, for example one set in GAIA’s environment. With--no-rememberit deletes an older stored token, so Claude models have no token instead of a stale one; set one of the variables above to use them.- AMD’s gateway only, unless you say otherwise. If you registered a different
gateway, set
GAIA_GATEWAY_ANTHROPIC_BASE_URLto its Anthropic base URL. GAIA won’t guess one, so your token can’t go to a host it wasn’t issued for. - No silent fallback. If the Anthropic route fails (bad token, a credential store
GAIA can’t read, model not deployed, off VPN, timeout), the turn fails with an error
that says what to fix. GAIA does not retry
through Lemonade, and a missing deployment is not recorded as
(unavailable). - Streaming works. Claude streams on this route, so the Models that cannot stream workaround never applies.
gaia gateway test --model amd.Claude-Opus-5-5tests this route too.
amd.DeepSeek-V4.1-Flash and amd.Gemma-4-31B included, still chats through Lemonade.
Models that cannot stream
For some models the gateway accepts a streaming request and then sends no tokens, while the same prompt works non-streaming.DeepSeek-V4.1-Flash and
Gemma-4-31B stream. Nothing in the catalog advertises this, so GAIA learns it the
first time it happens: it tells you the gateway did not stream that model,
answers the turn without streaming, and remembers the model in
~/.gaia/gateway.json so later turns skip straight to the working path. You see
one delayed first answer per model, then normal speed.
This is why the default is a model that streams: DeepSeek-V4.1-Flash, or
Gemma-4-31B when the gateway doesn’t offer it.
Native tool calling
The gateway’s catalog doesn’t label which models take tools natively (OpenAItools=), so the first time GAIA readies a gateway model it sends one tiny
request with a single tool and records the answer in ~/.gaia/gateway.json
(tool_calling_models / no_tool_calling_models). The probe gives up after
about 30 seconds. A probe that fails or proves nothing is logged and noted under
tool_calling_probe_failures, and is tried again an hour later rather than at
every start. Delete the model from those lists to re-probe it.
A model that can take tools natively still starts on GAIA’s in-text tool format,
which costs far less context per call: the full tool schemas add about 12K
tokens to every request. If the model writes tool calls in its own markup
instead (Claude’s <invoke> XML, for example), GAIA runs that call and
switches the model to native tool calls from its next request, for the rest of
that session only. Every new session starts on the in-text format again, so a
model that follows it, such as DeepSeek-V4.1-Flash, is not moved to the
costlier native format for good by one stray reply.
Claude models (amd.Claude-*) are the exception: they go through the
gateway’s Anthropic route, which takes native tool calls and caches the tool
schemas, so they get native calls from their first request.
Troubleshooting
What the gateway offers
76 models, verified live. Measured latency and throughput on a 120-word generation:Gemma-4-31B streams at 78 TPS with 1.8s to first token, and DeepSeek-V4.1-Flash
streams too. Streaming returns no tokens for some other models, while
non-streaming works for all of them — a gateway-side limitation, not a GAIA one.
Every chat model on this gateway is served over OpenAI Chat Completions, which is
why GAIA registers the provider with Lemonade using wire_format: openai. Claude
models are also served over the Anthropic Messages format, and GAIA uses that route
for them because it caches prompts. See
Claude models and prompt caching.
Disconnecting
logout does not unset LEMONADE_AMD_API_KEY — if that is set in Lemonade’s
environment, unset it there and restart Lemonade.