AMD LLM Gateway

What this gives you

GAIA normally runs models Lemonade loads on your own machine. The AMD LLM gateway hosts models too large for a laptop — a Gemma-4-31B-class on-prem model, and frontier models like Claude — behind one OpenAI-compatible endpoint. Connecting it means every GAIA agent can use those models with no code change: they appear in Lemonade’s model list alongside your local ones, and gaia chat, gaia llm, and the TUI address them the same way. That also makes it possible to run the same task through a GAIA agent and through another tool on the same model, and compare the results directly.

Requires Lemonade Server 11.8.0 or newer, which added cloud offload. Run gaia init to upgrade.

Where your token is kept

You enter it once. GAIA remembers it in your OS credential store — DPAPI on Windows, Keychain on macOS, SecretService on Linux — so it is encrypted at rest and scoped to your user account. GAIA refuses to fall back to a plaintext keyring backend, so it cannot quietly become a readable file. It is never written to a GAIA file. ~/.gaia/gateway.json records the base URL, which models you enabled, and what GAIA learned about them, and has no field a secret could occupy. Lemonade itself holds the token in memory only and forgets it on restart. GAIA notices and restores it from the credential store, which is why you are not asked again.

gaia gateway auth --no-remember    # do not store it; ask me every restart
gaia gateway logout                # clear Lemonade's copy, keep the remembered one
gaia gateway logout --forget       # also delete it from the credential store
gaia gateway status                # shows whether a token is remembered

LEMONADE_AMD_API_KEY in Lemonade’s environment still works and takes precedence. If it is set and you supply a different token, Lemonade refuses with a clear error rather than silently picking one.

Connect from the TUI

Type /gateway in chat (or pick it from the / command palette).

Confirm the gateway URL

The OpenAI-compatible base URL, e.g. https://llm-api.amd.com/Unified/v1. GAIA tests it before registering, so a wrong path fails immediately with the actual HTTP status instead of showing up later as an empty model list.

Paste your token

The field is masked. GAIA sends it to Lemonade and keeps no copy.

Done — a model is already selected

GAIA picks DeepSeek-V4.1-Flash for you and makes it the default. GAIA’s agent path streams, so the default has to be a model the gateway streams, and this one does; it is also the fastest model on GAIA’s agent benchmark. If your gateway doesn’t offer it, GAIA picks Gemma-4-31B, which streams too and is on-prem, so it costs nothing per token.To use a different one: space enables a model, enter makes it active. Preferred models are marked * and sorted first. A model whose last call returned “not found” is marked (unavailable) and is never picked for you.

The active model becomes the default for GAIA’s model-bearing commands.

Connect from the CLI

gaia gateway install                 # defaults to the AMD gateway
gaia gateway auth                    # prompts without echo
gaia gateway models                  # what the gateway offers
gaia gateway use amd.Claude-Opus-5   # make it the default
gaia gateway test "Say hello"        # end-to-end round trip on the active model
gaia gateway test --model amd.Gemma-4-31B "Say hello"   # ...or on a specific one

gaia gateway test names the model it is testing. If the gateway answers “not found” (HTTP 404) for it, the test names the model and a few deployed ones to switch to with gaia gateway use <id>. gaia gateway status shows registration, auth state, discovered model count, your current selection, and whether the active model’s last call failed.

When a model stops being served

Gateways retire and rename models. When a call to a gateway model returns HTTP 404, GAIA records it in ~/.gaia/gateway.json with the time, and a later successful call clears it. GAIA never switches to another model on its own; it tells you instead:

gaia chat, gaia llm, and gaia prompt print one warning line when your saved default model is the one that failed.
gaia gateway status and gaia gateway models mark it unavailable.
The TUI’s /gateway screen marks it (unavailable), says so under the list when it is the active model, and never auto-selects it.
A chat turn in the Agent UI or the TUI ends with an error that names the model and says retrying won’t help. The Agent UI points you to its model menu (or to Settings > Advanced > Custom model ID if you typed the ID there); the TUI points you to /model.

A 401 or 403 is an authentication problem, not a missing model, so it is not recorded.

Gateways with a different auth header

The default is Ocp-Apim-Subscription-Key with no prefix, because AMD’s gateway runs behind Azure API Management and that is the header it checks. So a plain OpenAI-compatible endpoint expecting a bearer token is the case that needs the override — not the other way round:

gaia gateway install \
  --base-url https://gateway.example.com/v1 \
  --auth-header-name Authorization \
  --auth-header-prefix "Bearer "

Both settings persist per provider and are re-applied on every request.

Using gateway models

Gateway models are namespaced amd.<model> and work anywhere a local model does:

gaia chat --model amd.Claude-Opus-5
gaia llm --model amd.Gemma-4-31B "Explain NPU offload"

GAIA recognises them as gateway-hosted and skips the download, load, and model-slot steps that only apply to local weights — so switching to a gateway model never evicts the model resident on your NPU or GPU. Tool-calling support comes from what the gateway advertises per model, so an agent will not try native tool calls against a model that cannot do them.

Models that cannot stream

For some models the gateway accepts a streaming request and then sends no tokens, while the same prompt works non-streaming. DeepSeek-V4.1-Flash and Gemma-4-31B stream. Nothing in the catalog advertises this, so GAIA learns it the first time it happens: it tells you the gateway did not stream that model, answers the turn without streaming, and remembers the model in ~/.gaia/gateway.json so later turns skip straight to the working path. You see one delayed first answer per model, then normal speed. This is why the default is a model that streams: DeepSeek-V4.1-Flash, or Gemma-4-31B when the gateway doesn’t offer it.

Troubleshooting

Symptom	Cause and fix
`models` lists nothing	Lemonade only discovers models once it has a working token. Run `gaia gateway auth`.
Setting a token returns a conflict	`LEMONADE_AMD_API_KEY` is set in Lemonade’s environment and wins. Keep using it, or unset it and restart Lemonade.
`GAIA's Lemonade Server is restarting and is not back yet`	GAIA’s own Lemonade was stopped and the background service has not brought it back yet, so the token was sent nowhere. Try again in a moment; if it persists, run `gaia daemon status`. GAIA never sends the token to another server on the default port. Set `LEMONADE_BASE_URL` to use a different server.
`has no cloud-provider API`	Lemonade is older than 11.8.0. Run `gaia init`.
The probe returns 404 or redirects	Wrong host. `llm.amd.com` is the SSO portal; the API is `llm-api.amd.com/Unified/v1`.
`auth` succeeds but discovers 0 models	The gateway rejected the key. Lemonade stores it without validating, so this is where a bad key first shows. Run `scripts/diagnose-gateway-auth.ps1`.
`The selected cloud model is unavailable (HTTP 404)`	The gateway no longer serves that model, or your account cannot use it. Run `gaia gateway models`, then `gaia gateway use <id>` with a deployed one. `gaia gateway test --model <id>` checks a model before you switch.
A gateway model answers all at once instead of streaming	Expected. The gateway does not stream that model, so GAIA switched to a non-streaming call and said so. See Models that cannot stream.

What the gateway offers

76 models, verified live. Measured latency and throughput on a 120-word generation:

Model	Provider	Latency	Avg TPS
`GPT-oss-20B`	On-Prem	3.1s	96
`Gemma-4-31B`	On-Prem	3.7s	40
`Qwen3.6-35B-A3B`	On-Prem	4.8s	62
`Claude-Opus-5`	Azure OpenAI	5.6s	53
`GPT-oss-120B`	On-Prem	8.4s	31

Gemma-4-31B streams at 78 TPS with 1.8s to first token, and DeepSeek-V4.1-Flash streams too. Streaming returns no tokens for some other models, while non-streaming works for all of them — a gateway-side limitation, not a GAIA one. Every chat model on this gateway — Claude included — is served over OpenAI Chat Completions, not the Anthropic Messages format. That is why GAIA registers the provider with wire_format: openai and why Claude models work through the ordinary agent path with no special handling.

Disconnecting

gaia gateway logout      # drop the session token, keep the registration
gaia gateway uninstall   # remove the provider and forget the selection

logout does not unset LEMONADE_AMD_API_KEY — if that is set in Lemonade’s environment, unset it there and restart Lemonade.

Getting Started

C++ Framework

Python Framework

What this gives you

Where your token is kept

Connect from the TUI

Connect from the CLI

When a model stops being served

Gateways with a different auth header

Using gateway models

Models that cannot stream

Troubleshooting

What the gateway offers

Disconnecting

​What this gives you

​Where your token is kept

​Connect from the TUI

​Connect from the CLI

​When a model stops being served

​Gateways with a different auth header

​Using gateway models

​Models that cannot stream

​Troubleshooting

​What the gateway offers

​Disconnecting

What this gives you

Where your token is kept

Connect from the TUI

Connect from the CLI

When a model stops being served

Gateways with a different auth header

Using gateway models

Models that cannot stream

Troubleshooting

What the gateway offers

Disconnecting