> ## Documentation Index > Fetch the complete documentation index at: https://amd-gaia.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # AMD LLM Gateway > Run GAIA agents on gateway-hosted models — larger on-prem models and frontier models — alongside local inference. ## What this gives you GAIA normally runs models Lemonade loads on your own machine. The AMD LLM gateway hosts models too large for a laptop — a Gemma-4-31B-class on-prem model, and frontier models like Claude — behind one OpenAI-compatible endpoint. Connecting it means **every GAIA agent can use those models with no code change**: they appear in Lemonade's model list alongside your local ones, and `gaia chat`, `gaia llm`, and the TUI address them the same way. That also makes it possible to run the same task through a GAIA agent and through another tool on the same model, and compare the results directly. > [!NOTE] > > Requires **Lemonade Server 11.8.0 or newer**, which added cloud offload. Run > `gaia init` to upgrade. ## Where your token is kept You enter it once. GAIA remembers it in your **OS credential store** — DPAPI on Windows, Keychain on macOS, SecretService on Linux — so it is encrypted at rest and scoped to your user account. GAIA refuses to fall back to a plaintext keyring backend, so it cannot quietly become a readable file. It is never written to a GAIA file. `~/.gaia/gateway.json` records the base URL, which models you enabled, and what GAIA learned about them, and has no field a secret could occupy. Lemonade itself holds the token in memory only and forgets it on restart. GAIA notices and restores it from the credential store, which is why you are not asked again. ```bash theme={null} gaia gateway auth --no-remember # do not store it; ask me every restart gaia gateway logout # clear Lemonade's copy, keep the remembered one gaia gateway logout --forget # also delete it from the credential store gaia gateway status # shows whether a token is remembered ``` `LEMONADE_AMD_API_KEY` in **Lemonade's** environment still works and takes precedence. If it is set and you supply a different token, Lemonade refuses with a clear error rather than silently picking one. ## Connect from the TUI Type `/gateway` in chat (or pick it from the `/` command palette). **1. Confirm the gateway URL** The OpenAI-compatible base URL, e.g. `https://llm-api.amd.com/Unified/v1`. GAIA tests it before registering, so a wrong path fails immediately with the actual HTTP status instead of showing up later as an empty model list. **2. Paste your token** The field is masked. GAIA sends it to Lemonade and keeps no copy. **3. Done — a model is already selected** GAIA picks `DeepSeek-V4.1-Flash` for you and makes it the default. GAIA's agent path streams, so the default has to be a model the gateway streams, and this one does; it is also the fastest model on GAIA's agent benchmark. If your gateway doesn't offer it, GAIA picks `Gemma-4-31B`, which streams too and is on-prem, so it costs nothing per token. To use a different one: space enables a model, enter makes it active. Preferred models are marked `*` and sorted first. A model whose last call returned "not found" is marked `(unavailable)` and is never picked for you. The active model becomes the default for GAIA's model-bearing commands. ## Connect from the CLI ```bash theme={null} gaia gateway install # defaults to the AMD gateway gaia gateway auth # prompts without echo gaia gateway models # what the gateway offers gaia gateway use amd.Claude-Opus-5 # make it the default gaia gateway test "Say hello" # end-to-end round trip on the active model gaia gateway test --model amd.Gemma-4-31B "Say hello" # ...or on a specific one ``` `gaia gateway test` names the model it is testing. If the gateway answers "not found" (HTTP 404) for it, the test names the model and a few deployed ones to switch to with `gaia gateway use `. `gaia gateway status` shows registration, auth state, discovered model count, your current selection, and whether the active model's last call failed. ### When a model stops being served Gateways retire and rename models. When a call to a gateway model returns HTTP 404, GAIA records it in `~/.gaia/gateway.json` with the time, and a later successful call clears it. GAIA never switches to another model on its own; it tells you instead: * `gaia chat`, `gaia llm`, and `gaia prompt` print one warning line when your saved default model is the one that failed. * `gaia gateway status` and `gaia gateway models` mark it unavailable. * The TUI's `/gateway` screen marks it `(unavailable)`, says so under the list when it is the active model, and never auto-selects it. * A chat turn in the Agent UI or the TUI ends with an error that names the model and says retrying won't help. The Agent UI points you to its model menu (or to **Settings > Advanced > Custom model ID** if you typed the ID there); the TUI points you to `/model`. A 401 or 403 is an authentication problem, not a missing model, so it is not recorded. ### Gateways with a different auth header The default is `Ocp-Apim-Subscription-Key` with **no** prefix, because AMD's gateway runs behind Azure API Management and that is the header it checks. So a plain OpenAI-compatible endpoint expecting a bearer token is the case that needs the override — not the other way round: ```bash theme={null} gaia gateway install \ --base-url https://gateway.example.com/v1 \ --auth-header-name Authorization \ --auth-header-prefix "Bearer " ``` Both settings persist per provider and are re-applied on every request. ## Using gateway models Gateway models are namespaced `amd.` and work anywhere a local model does: ```bash theme={null} gaia chat --model amd.Claude-Opus-5 gaia llm --model amd.Gemma-4-31B "Explain NPU offload" ``` GAIA recognises them as gateway-hosted and skips the download, load, and model-slot steps that only apply to local weights — so switching to a gateway model never evicts the model resident on your NPU or GPU. Tool-calling support comes from what the gateway advertises per model, so an agent will not try native tool calls against a model that cannot do them. ### Models that cannot stream For some models the gateway accepts a streaming request and then sends no tokens, while the same prompt works non-streaming. `DeepSeek-V4.1-Flash` and `Gemma-4-31B` stream. Nothing in the catalog advertises this, so GAIA learns it the first time it happens: it tells you the gateway did not stream that model, answers the turn without streaming, and remembers the model in `~/.gaia/gateway.json` so later turns skip straight to the working path. You see one delayed first answer per model, then normal speed. This is why the default is a model that streams: `DeepSeek-V4.1-Flash`, or `Gemma-4-31B` when the gateway doesn't offer it. ## Troubleshooting | Symptom | Cause and fix | | - | - | | `models` lists nothing | Lemonade only discovers models once it has a working token. Run `gaia gateway auth`. | | Setting a token returns a conflict | `LEMONADE_AMD_API_KEY` is set in Lemonade's environment and wins. Keep using it, or unset it and restart Lemonade. | | `GAIA's Lemonade Server is restarting and is not back yet` | GAIA's own Lemonade was stopped and the background service has not brought it back yet, so the token was sent nowhere. Try again in a moment; if it persists, run `gaia daemon status`. GAIA never sends the token to another server on the default port. Set `LEMONADE_BASE_URL` to use a different server. | | `has no cloud-provider API` | Lemonade is older than 11.8.0. Run `gaia init`. | | The probe returns 404 or redirects | Wrong host. `llm.amd.com` is the SSO portal; the API is `llm-api.amd.com/Unified/v1`. | | `auth` succeeds but discovers 0 models | The gateway rejected the key. Lemonade stores it without validating, so this is where a bad key first shows. Run `scripts/diagnose-gateway-auth.ps1`. | | `The selected cloud model is unavailable (HTTP 404)` | The gateway no longer serves that model, or your account cannot use it. Run `gaia gateway models`, then `gaia gateway use ` with a deployed one. `gaia gateway test --model ` checks a model before you switch. | | A gateway model answers all at once instead of streaming | Expected. The gateway does not stream that model, so GAIA switched to a non-streaming call and said so. See [Models that cannot stream](#models-that-cannot-stream). | ## What the gateway offers 76 models, verified live. Measured latency and throughput on a 120-word generation: | Model | Provider | Latency | Avg TPS | | - | - | - | - | | `GPT-oss-20B` | On-Prem | 3.1s | 96 | | `Gemma-4-31B` | On-Prem | 3.7s | 40 | | `Qwen3.6-35B-A3B` | On-Prem | 4.8s | 62 | | `Claude-Opus-5` | Azure OpenAI | 5.6s | 53 | | `GPT-oss-120B` | On-Prem | 8.4s | 31 | `Gemma-4-31B` streams at 78 TPS with 1.8s to first token, and `DeepSeek-V4.1-Flash` streams too. **Streaming returns no tokens for some other models**, while non-streaming works for all of them — a gateway-side limitation, not a GAIA one. **Every chat model on this gateway — Claude included — is served over OpenAI Chat Completions**, not the Anthropic Messages format. That is why GAIA registers the provider with `wire_format: openai` and why Claude models work through the ordinary agent path with no special handling. ## Disconnecting ```bash theme={null} gaia gateway logout # drop the session token, keep the registration gaia gateway uninstall # remove the provider and forget the selection ``` `logout` does not unset `LEMONADE_AMD_API_KEY` — if that is set in Lemonade's environment, unset it there and restart Lemonade. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.