Skip to main content

What this gives you

GAIA normally runs models Lemonade loads on your own machine. The AMD LLM gateway hosts models too large for a laptop — a Gemma-4-31B-class on-prem model, and frontier models like Claude — behind one OpenAI-compatible endpoint. Connecting it means every GAIA agent can use those models with no code change: they appear in Lemonade’s model list alongside your local ones, and gaia chat, gaia llm, and the TUI address them the same way. That also makes it possible to run the same task through a GAIA agent and through another tool on the same model, and compare the results directly.

Where your token is kept

You enter it once. GAIA remembers it in your OS credential store — DPAPI on Windows, Keychain on macOS, SecretService on Linux — so it is encrypted at rest and scoped to your user account. GAIA refuses to fall back to a plaintext keyring backend, so it cannot quietly become a readable file. It is never written to a GAIA file. ~/.gaia/gateway.json records the base URL, which models you enabled, and what GAIA learned about them, and has no field a secret could occupy. Lemonade itself holds the token in memory only and forgets it on restart. GAIA notices and restores it from the credential store, or from LEMONADE_AMD_API_KEY / GAIA_GATEWAY_TOKEN in GAIA’s environment when the gateway is AMD’s, which is why you are not asked again. gaia gateway install restores it too. Installing with a different --base-url deletes the remembered token instead, so a token for one gateway is never sent to another; run gaia gateway auth for the new one.
LEMONADE_AMD_API_KEY in Lemonade’s environment still works and takes precedence. If it is set and you supply a different token, Lemonade refuses with a clear error rather than silently picking one. Claude models read the token from GAIA, not Lemonade, so set the same variable in GAIA’s environment too; see Claude models and prompt caching.

Connect from the TUI

Type /gateway in chat (or pick it from the / command palette).
1

Confirm the gateway URL

The OpenAI-compatible base URL, e.g. https://llm-api.amd.com/Unified/v1. GAIA tests it before registering, so a wrong path fails immediately with the actual HTTP status instead of showing up later as an empty model list.
2

Paste your token

The field is masked. GAIA sends it to Lemonade and keeps no copy.
3

Done — a model is already selected

GAIA picks DeepSeek-V4.1-Flash for you and makes it the default. GAIA’s agent path streams, so the default has to be a model the gateway streams, and this one does; it is also the fastest model on GAIA’s agent benchmark. If your gateway doesn’t offer it, GAIA picks Gemma-4-31B, which streams too and is on-prem, so it costs nothing per token.To use a different one: space enables a model, enter makes it active. Preferred models are marked * and sorted first. A model whose last call returned “not found” is marked (unavailable) and is never picked for you.
The active model becomes the default for GAIA’s model-bearing commands.

Connect from the CLI

gaia gateway test names the model it is testing. If the gateway answers “not found” (HTTP 404) for it, the test names the model and a few deployed ones to switch to with gaia gateway use <id>. gaia gateway status shows registration, auth state, discovered model count, your current selection, and whether the active model’s last call failed.

When a model stops being served

Gateways retire and rename models. When a call to a gateway model returns HTTP 404, GAIA records it in ~/.gaia/gateway.json with the time, and a later successful call clears it. GAIA never switches to another model on its own; it tells you instead:
  • gaia chat, gaia llm, and gaia prompt print one warning line when your saved default model is the one that failed.
  • gaia gateway status and gaia gateway models mark it unavailable.
  • The TUI’s /gateway screen marks it (unavailable), says so under the list when it is the active model, and never auto-selects it.
  • A chat turn in the Agent UI or the TUI ends with an error that names the model and says retrying won’t help. The Agent UI points you to its model menu (or to Settings > Advanced > Custom model ID if you typed the ID there); the TUI points you to /model.
A 401 or 403 is an authentication problem, not a missing model, so it is not recorded.

Gateways with a different auth header

The default is Ocp-Apim-Subscription-Key with no prefix, because AMD’s gateway runs behind Azure API Management and that is the header it checks. So a plain OpenAI-compatible endpoint expecting a bearer token is the case that needs the override — not the other way round:
Both settings persist per provider and are re-applied on every request.

Using gateway models

Gateway models are namespaced amd.<model> and work anywhere a local model does:
GAIA recognises them as gateway-hosted and skips the download, load, and model-slot steps that only apply to local weights — so switching to a gateway model never evicts the model resident on your NPU or GPU. Tool-calling support comes from what the gateway advertises per model, so an agent will not try native tool calls against a model that cannot do them.

Claude models and prompt caching

Chat with a Claude model (amd.Claude-*) skips Lemonade and goes straight to the gateway’s Anthropic Messages route, https://llm-api.amd.com/Anthropic — the same route Claude Code uses. The OpenAI-compatible route Lemonade calls ignores Anthropic’s cache markers, so every agent step re-billed the whole prompt. On the Anthropic route, the tool schemas, system prompt, and conversation so far are read from cache on later steps: a repeated 48K-token prompt came back with all but 4 tokens cached. gaia -p --output-format json reports this as cached_tokens. What that means in practice:
  • Same token as every other gateway model. GAIA sends it in the Ocp-Apim-Subscription-Key header itself, reading LEMONADE_AMD_API_KEY, then GAIA_GATEWAY_TOKEN in GAIA’s environment, then the OS credential store that gaia gateway auth writes to. When Lemonade forgets its token, GAIA gives it back this same one, so Claude and DeepSeek can’t end up on different keys. GAIA never sends an Anthropic API key there.
  • Lemonade gets the token in GAIA’s environment only for AMD’s gateway. If you registered a different gateway, GAIA gives Lemonade only the token gaia gateway auth stored for it, and logs a warning that the environment token was left out. Installing with a different --base-url deletes the stored token, so run gaia gateway auth for the new gateway.
  • Lemonade’s own LEMONADE_AMD_API_KEY should be in GAIA’s environment too. When Lemonade was started with that variable, it ignores tokens sent at runtime, and GAIA can’t read the key back. Claude models then send GAIA’s other copy (GAIA_GATEWAY_TOKEN or a stored token) and log a warning that the two can drift. If that copy has gone stale, the 401 says so. Set LEMONADE_AMD_API_KEY to the same key in GAIA’s environment to make them one.
  • gaia gateway auth updates both. It fails if Claude models would still send a different token, for example one set in GAIA’s environment. With --no-remember it deletes an older stored token, so Claude models have no token instead of a stale one; set one of the variables above to use them.
  • AMD’s gateway only, unless you say otherwise. If you registered a different gateway, set GAIA_GATEWAY_ANTHROPIC_BASE_URL to its Anthropic base URL. GAIA won’t guess one, so your token can’t go to a host it wasn’t issued for.
  • No silent fallback. If the Anthropic route fails (bad token, a credential store GAIA can’t read, model not deployed, off VPN, timeout), the turn fails with an error that says what to fix. GAIA does not retry through Lemonade, and a missing deployment is not recorded as (unavailable).
  • Streaming works. Claude streams on this route, so the Models that cannot stream workaround never applies.
  • gaia gateway test --model amd.Claude-Opus-5-5 tests this route too.
Lemonade still has to be running with the gateway connected: agent startup checks it and probes tool calling through it. Every other gateway model, amd.DeepSeek-V4.1-Flash and amd.Gemma-4-31B included, still chats through Lemonade.

Models that cannot stream

For some models the gateway accepts a streaming request and then sends no tokens, while the same prompt works non-streaming. DeepSeek-V4.1-Flash and Gemma-4-31B stream. Nothing in the catalog advertises this, so GAIA learns it the first time it happens: it tells you the gateway did not stream that model, answers the turn without streaming, and remembers the model in ~/.gaia/gateway.json so later turns skip straight to the working path. You see one delayed first answer per model, then normal speed. This is why the default is a model that streams: DeepSeek-V4.1-Flash, or Gemma-4-31B when the gateway doesn’t offer it.

Native tool calling

The gateway’s catalog doesn’t label which models take tools natively (OpenAI tools=), so the first time GAIA readies a gateway model it sends one tiny request with a single tool and records the answer in ~/.gaia/gateway.json (tool_calling_models / no_tool_calling_models). The probe gives up after about 30 seconds. A probe that fails or proves nothing is logged and noted under tool_calling_probe_failures, and is tried again an hour later rather than at every start. Delete the model from those lists to re-probe it. A model that can take tools natively still starts on GAIA’s in-text tool format, which costs far less context per call: the full tool schemas add about 12K tokens to every request. If the model writes tool calls in its own markup instead (Claude’s <invoke> XML, for example), GAIA runs that call and switches the model to native tool calls from its next request, for the rest of that session only. Every new session starts on the in-text format again, so a model that follows it, such as DeepSeek-V4.1-Flash, is not moved to the costlier native format for good by one stray reply. Claude models (amd.Claude-*) are the exception: they go through the gateway’s Anthropic route, which takes native tool calls and caches the tool schemas, so they get native calls from their first request.

Troubleshooting

What the gateway offers

76 models, verified live. Measured latency and throughput on a 120-word generation: Gemma-4-31B streams at 78 TPS with 1.8s to first token, and DeepSeek-V4.1-Flash streams too. Streaming returns no tokens for some other models, while non-streaming works for all of them — a gateway-side limitation, not a GAIA one. Every chat model on this gateway is served over OpenAI Chat Completions, which is why GAIA registers the provider with Lemonade using wire_format: openai. Claude models are also served over the Anthropic Messages format, and GAIA uses that route for them because it caches prompts. See Claude models and prompt caching.

Disconnecting

logout does not unset LEMONADE_AMD_API_KEY — if that is set in Lemonade’s environment, unset it there and restart Lemonade.