LLM Client - GAIA SDK

Source Code:

src/gaia/llm/__init__.py - Package exports
src/gaia/llm/base_client.py - Abstract interface
src/gaia/llm/factory.py - Client factory
src/gaia/llm/providers/ - Provider implementations

Primary API: create_client() factory function Module: gaia.llm Imports:

from gaia.llm import create_client (preferred)
from gaia.llm import LLMClient, NotSupportedError

Overview

The LLM client package provides a unified interface for generating text from multiple LLM backends using a provider pattern. Each provider implements the abstract LLMClient interface, with optional methods raising NotSupportedError when unavailable. Key Features:

Factory-based client creation with create_client()
Two providers: Lemonade (local AMD-optimized, plus models configured behind Lemonade such as on-prem gateways) and Claude
Abstract base class for type safety and extensibility
Graceful handling of unsupported features via NotSupportedError
Streaming and non-streaming generation
Backward-compatible use_claude flag

Provider Capabilities:

Method	Lemonade	Claude
`generate()`	✓	✓
`chat()`	✓	✓
`embed()`	✓	✗
`vision()`	✓	✓
`get_performance_stats()`	✓	✓
`load_model()`	✓	✗
`unload_model()`	✓	✗

Methods marked with ✗ raise NotSupportedError when called on that provider. Cloud models served through Lemonade have their own capabilities; selecting Lemonade does not imply that every remote model supports every operation.

Removed adapters and migration

The direct openai and litellm adapters were removed because they discarded native tool calls. The litellm installation extra is removed too. The openai Python dependency remains: Lemonade uses it for compatible HTTP requests. provider="openai", provider="litellm", and use_openai=True raise ValueError with migration guidance before importing a provider or issuing requests. use_chatgpt=True in the agent, chat/audio/talk SDK chain and the legacy --use-chatgpt CLI flag also fail explicitly. These flags are retained only as migration-error shims; they do not select another backend or ignore the setting. Configure the upstream model in Lemonade, then pass the Lemonade server URL and its catalog model ID. Do not substitute a raw upstream endpoint for Lemonade’s URL: a generic OpenAI-compatible service does not implement Lemonade’s catalog and lifecycle API. See gateway migration. Historical release notes describe the providers that shipped in those releases.

Requirements

Functional Requirements

Factory Pattern
- create_client() factory function for client creation
- Explicit provider selection via provider parameter (“lemonade”, “claude”)
- Backward-compatible use_claude flag
- Auto-detection of provider from flags when provider not specified
- Default to Lemonade provider when no flags set
- Retired names ("openai", "litellm") and a true use_openai flag raise ValueError with migration guidance before any provider is imported
Abstract Interface
- LLMClient ABC defines unified interface
- provider_name property returns provider name
- Required methods (all providers must implement):
  - generate() - Text completion
  - chat() - Chat completion with message history
- Optional methods (raise NotSupportedError if not implemented):
  - embed() - Generate embeddings
  - vision() - Vision/image understanding
  - get_performance_stats() - Performance statistics
  - load_model() - Load a model
  - unload_model() - Unload current model
Provider Implementations
- LemonadeProvider: Full support for all methods, connects to local Lemonade server
- ClaudeProvider: generate, chat, vision, get_performance_stats (last-call usage)
- All providers support streaming and non-streaming modes
Error Handling
- NotSupportedError raised for unsupported methods
- Clear error messages indicating provider and unsupported method
- Connection errors handled by underlying provider implementations

Non-Functional Requirements

Performance
- Lazy provider loading via importlib (load only when needed)
- Minimal overhead from abstraction layer
- Streaming support across all providers
- Local models run with their model card’s sampling, else temperature 0.1 with penalties; cloud models keep the client default
Reliability
- Type safety through ABC pattern
- Graceful handling of unsupported features
- Clear error messages for provider capabilities
- Provider-specific connection management
Usability
- Simple factory function interface
- Backward compatibility with existing code
- Consistent API across all providers
- Clear documentation with examples

API Specification

Package Structure

src/gaia/llm/
├── __init__.py              # Package exports
├── base_client.py           # Abstract LLMClient interface
├── factory.py               # create_client() factory function
├── exceptions.py            # NotSupportedError
├── lemonade_client.py       # Low-level REST client for Lemonade
└── providers/
    ├── lemonade.py          # LemonadeProvider
    └── claude.py            # ClaudeProvider

Package Exports (`init.py`)

from gaia.llm import create_client, LLMClient, VLMClient, NotSupportedError

VLMClient is also re-exported from gaia.llm — see vlm-client for its API.

Factory Function (`factory.py`)

def create_client(
    provider: Optional[str] = None,
    use_claude: bool = False,
    use_openai: bool = False,
    **kwargs,
) -> LLMClient:
    """
    Create a Lemonade or Claude client.

    Args:
        provider: Explicit provider name ("lemonade" or "claude").
                  If not specified, auto-detected from the use_claude flag.
        use_claude: If True, use Claude provider (ignored if provider is specified)
        use_openai: Retained only to reject legacy callers; True always raises
        **kwargs: Provider-specific arguments (base_url, model, api_key, etc.)

    Returns:
        LLMClient instance for the specified or detected provider

    Raises:
        ValueError: If provider is unknown, is a retired name ("openai",
                    "litellm"), or use_openai is True

    Examples:
        # Default Lemonade provider
        client = create_client()

        # Explicit provider selection
        client = create_client(provider="lemonade", model="Qwen3-0.6B-GGUF")
        client = create_client(provider="claude", api_key="sk-ant-...")

        # Backward-compatible flag
        client = create_client(use_claude=True, api_key="sk-ant-...")

    Note:
        Provider defaults to "lemonade" when no flags are set.
        An explicit supported provider takes precedence over use_claude.
    """

Abstract Base Class (`base_client.py`)

from abc import ABC, abstractmethod
from typing import Iterator, Union

class LLMClient(ABC):
    """
    Unified LLM client interface.

    Methods raise NotSupportedError if not available for this provider.
    """

    @property
    @abstractmethod
    def provider_name(self) -> str:
        """Return the provider name for error messages."""
        ...

    @abstractmethod
    def generate(
        self,
        prompt: str,
        model: str | None = None,
        stream: bool = False,
        **kwargs,
    ) -> Union[str, Iterator[str]]:
        """
        Generate text completion.

        Args:
            prompt: The user prompt/query to send to the LLM
            model: The model to use (defaults to provider's default model)
            stream: If True, returns a generator that yields chunks of the response
            **kwargs: Additional parameters (temperature, max_tokens, etc.)

        Returns:
            If stream=False: The complete generated text as a string
            If stream=True: A generator yielding chunks of the response

        Example:
            response = client.generate("Write a hello world program")
        """
        ...

    @abstractmethod
    def chat(
        self,
        messages: list[dict],
        model: str | None = None,
        stream: bool = False,
        **kwargs,
    ) -> Union[str, Iterator[str]]:
        """
        Chat completion with message history.

        Args:
            messages: List of message dicts with 'role' and 'content' keys
            model: The model to use (defaults to provider's default model)
            stream: If True, returns a generator that yields chunks of the response
            **kwargs: Additional parameters (temperature, max_tokens, etc.)

        Returns:
            If stream=False: The complete generated text as a string
            If stream=True: A generator yielding chunks of the response

        Example:
            messages = [
                {"role": "user", "content": "Hello"},
                {"role": "assistant", "content": "Hi there!"},
                {"role": "user", "content": "How are you?"}
            ]
            response = client.chat(messages)
        """
        ...

    # Optional methods - default raises NotSupportedError
    def embed(self, texts: list[str], **kwargs) -> list[list[float]]:
        """
        Generate embeddings for texts.

        Args:
            texts: List of text strings to embed
            **kwargs: Additional parameters

        Returns:
            List of embedding vectors (list of floats)

        Raises:
            NotSupportedError: If provider doesn't support embeddings

        Note:
            Supported by: Lemonade
            Not supported by: Claude
        """
        raise NotSupportedError(self.provider_name, "embed")

    def vision(self, images: list[bytes], prompt: str, **kwargs) -> str:
        """
        Vision/image understanding.

        Args:
            images: List of image data as bytes
            prompt: Text prompt describing what to analyze
            **kwargs: Additional parameters

        Returns:
            Text response describing the image

        Raises:
            NotSupportedError: If provider doesn't support vision

        Note:
            Supported by: Lemonade, Claude
        """
        raise NotSupportedError(self.provider_name, "vision")

    def get_performance_stats(self) -> dict:
        """
        Get performance statistics from the last LLM request.

        Returns:
            Dictionary containing performance statistics

        Raises:
            NotSupportedError: If provider doesn't support performance stats

        Note:
            Supported by: Lemonade, Claude (last-call token usage)
        """
        raise NotSupportedError(self.provider_name, "get_performance_stats")

    def load_model(self, model_name: str, **kwargs) -> None:
        """
        Load a specific model.

        Args:
            model_name: Name of the model to load
            **kwargs: Additional parameters

        Raises:
            NotSupportedError: If provider doesn't support model loading

        Note:
            Only supported by: Lemonade
        """
        raise NotSupportedError(self.provider_name, "load_model")

    def unload_model(self) -> None:
        """
        Unload the current model.

        Raises:
            NotSupportedError: If provider doesn't support model unloading

        Note:
            Only supported by: Lemonade
        """
        raise NotSupportedError(self.provider_name, "unload_model")

NotSupportedError (`exceptions.py`)

class NotSupportedError(Exception):
    """Raised when a provider doesn't support a method."""

    def __init__(self, provider: str, method: str):
        super().__init__(f"{provider} does not support {method}")

Provider Implementations

LemonadeProvider (`providers/lemonade.py`)

Full feature support - implements all methods.

class LemonadeProvider(LLMClient):
    """Lemonade provider - local AMD-optimized inference."""

    def __init__(
        self,
        model: Optional[str] = None,
        base_url: Optional[str] = None,
        host: Optional[str] = None,
        port: Optional[int] = None,
        system_prompt: Optional[str] = None,
        **kwargs,
    ):
        """
        Initialize Lemonade provider.

        Args:
            model: Model name (defaults to DEFAULT_MODEL_NAME, currently "Gemma-4-E4B-it-GGUF")
            base_url: Base URL for Lemonade server (overrides LEMONADE_BASE_URL env var)
            host: Server host (alternative to base_url)
            port: Server port (alternative to base_url)
            system_prompt: Default system prompt for chat
            api_key: API key for an authenticated Lemonade server
                     (overrides LEMONADE_API_KEY env var; None for unauthenticated)
            **kwargs: Additional arguments passed to LemonadeClient

        Environment:
            LEMONADE_BASE_URL: Default base URL (http://localhost:13305/api/v1)
            LEMONADE_API_KEY: API key for authenticated remote Lemonade
                              (sent as Authorization: Bearer header on every request)
            LEMONADE_MODEL: Default model name if not specified

        Note:
            Default model is DEFAULT_MODEL_NAME (currently "Gemma-4-E4B-it-GGUF").
            Local models use their model card's sampling, else temperature=0.1; cloud models keep the client default.
        """

    # Supports all methods: generate, chat, embed, vision,
    # get_performance_stats, load_model, unload_model

ClaudeProvider (`providers/claude.py`)

Partial support - generate, chat, vision, get_performance_stats.

class ClaudeProvider(LLMClient):
    """Claude (Anthropic) provider."""

    def __init__(
        self,
        api_key: Optional[str] = None,
        model: str = DEFAULT_CLAUDE_MODEL,  # "claude-sonnet-5"
        system_prompt: Optional[str] = None,
        max_retries: int = 3,
        timeout: float = 300.0,
        **_kwargs,
    ):
        """
        Initialize Claude provider.

        Args:
            api_key: Anthropic API key (defaults to ANTHROPIC_API_KEY env var)
            model: Model name (default: DEFAULT_CLAUDE_MODEL)
            system_prompt: Default system prompt for chat

        Environment:
            ANTHROPIC_API_KEY: API key for Anthropic Claude

        Raises:
            ImportError: If anthropic package not installed
            ValueError: If no API key is available
        """

    # Supports: generate, chat, vision, get_performance_stats
    # Raises NotSupportedError: embed, load_model, unload_model

Implementation Details

Provider Selection Logic

The factory function auto-detects the provider based on parameters:

# From factory.py
def create_client(provider=None, use_claude=False, use_openai=False, **kwargs):
    # Retired selections fail before any provider import
    if use_openai or (provider and provider.lower() in {"openai", "litellm"}):
        raise ValueError(REMOVED_PROVIDER_MESSAGE)
    if provider is None:
        provider = "claude" if use_claude else "lemonade"

    # Validate provider
    if provider.lower() not in _PROVIDERS:
        available = ", ".join(_PROVIDERS.keys())
        raise ValueError(f"Unknown provider: {provider}. Available: {available}")

    # Load provider class dynamically...

Lazy Provider Loading

Providers are loaded dynamically using importlib to avoid importing unnecessary dependencies:

_PROVIDERS = {
    "lemonade": "gaia.llm.providers.lemonade.LemonadeProvider",
    "claude": "gaia.llm.providers.claude.ClaudeProvider",
}

# Lazy import - only load when needed
module_path, class_name = _PROVIDERS[provider_lower].rsplit(".", 1)
module = importlib.import_module(module_path)
provider_class = getattr(module, class_name)

return provider_class(**kwargs)

NotSupportedError Pattern

Optional methods raise NotSupportedError by default in the ABC:

# In base_client.py
class LLMClient(ABC):
    def embed(self, texts: list[str], **kwargs) -> list[list[float]]:
        raise NotSupportedError(self.provider_name, "embed")

    def vision(self, images: list[bytes], prompt: str, **kwargs) -> str:
        raise NotSupportedError(self.provider_name, "vision")
    # etc...

Providers override only the methods they support:

# ClaudeProvider overrides vision but not embed
class ClaudeProvider(LLMClient):
    def vision(self, images: list[bytes], prompt: str, **kwargs) -> str:
        # Sends the image as a base64 content block
        ...

    # embed() inherited - raises NotSupportedError

Sampling Defaults

A local model whose card publishes sampling settings runs with them. These live in MODEL_SAMPLING_PROFILES in lemonade_client.py, keyed by Lemonade model id. For example, Qwen3-30B-A3B-Instruct-2507-GGUF gets temperature=0.7, top_p=0.8, top_k=20, min_p=0 and presence_penalty=1.0, per its model card. A card that gives separate thinking and non-thinking profiles registers both. The thinking mode comes from requested_thinking(): the caller’s chat_template_kwargs={"enable_thinking": ...} when it sets one, else GAIA’s choice for the model (ModelRequirement.thinking). The provider picks the sampling profile from it and LemonadeClient.chat_completions sends it as the switch, so the sampling always matches the mode the model runs in. Qwen3.6-35B-A3B-GGUF runs with thinking on and gets its card’s thinking profile for coding and precise work. Memory’s short JSON side calls (extraction, consolidation, reconciliation, skill distillation) pass no_thinking_kwargs(model), which turns thinking off for such a model: reasoning is billed against max_tokens, so with it on they reason to the cap and return no answer. Every other local model gets LOCAL_SAMPLING_DEFAULTS: temperature=0.1 plus frequency, presence and repeat penalties, so it does not loop on tables and paragraphs. Cloud models get neither, because near-greedy sampling sends a reasoning model’s thinking into a runaway. They keep the client default (0.7). Any sampling argument the caller passes is sent as given.

# In LemonadeProvider.chat()
if not self._backend.cloud_model_provider(effective_model):
    defaults = local_sampling_defaults(
        effective_model,
        requested_thinking(effective_model, kwargs.get("chat_template_kwargs")),
    )
    for key, value in defaults.items():
        kwargs.setdefault(key, value)

top_k, min_p, repeat_penalty and repeat_last_n are llama.cpp-native. When streaming, they travel in the OpenAI SDK’s extra_body and land at the top level of the request body.

Provider-Specific Implementation

LemonadeProvider wraps the low-level LemonadeClient:

class LemonadeProvider(LLMClient):
    def __init__(self, model=None, base_url=None, **kwargs):
        self._backend = LemonadeClient(model=model, base_url=base_url, **kwargs)

    def generate(self, prompt, model=None, stream=False, **kwargs):
        # Routed through chat (completions endpoint not available in Lemonade v9.1+)
        return self.chat([{"role": "user", "content": prompt}], model=model, stream=stream, **kwargs)

ClaudeProvider uses the Anthropic SDK:

class ClaudeProvider(LLMClient):
    def __init__(self, api_key=None, model=DEFAULT_CLAUDE_MODEL, **kwargs):
        import anthropic
        self._client = anthropic.Anthropic(api_key=api_key)
        self._model = model

Native tool-call transport

The shared agent consumes the provider’s string response. Lemonade preserves native calls in the __tool_calls__ sentinel envelope, including call IDs, function names and arguments; it does not replace a tool-only response with empty assistant text. Streaming deltas are accumulated before dispatch so a partial argument fragment cannot execute as a complete call. The agent turns results back into the corresponding tool messages for the next model request. Provider usage is available through get_last_usage() when the backend reports it. Missing usage is not proof of zero tokens or zero cost. Per-turn timing and usage records are owned by agents/base/turn_metrics.py.

Error boundaries

Configuration errors fail before inference. Lemonade transport errors are classified by the Lemonade provider into typed failures with user-facing guidance. Authentication and upstream service failures must remain attributable to the configured destination; no implicit retry with a different model or provider is permitted. Optional dependencies must not be required merely to reject a removed provider name.

Testing Requirements

Unit Tests

File: tests/unit/test_llm_client_factory.py

import pytest
from unittest.mock import patch, Mock
from gaia.llm import create_client, LLMClient, NotSupportedError

class TestImports:
    def test_can_import_create_client(self):
        """Verify create_client can be imported."""
        from gaia.llm import create_client
        assert callable(create_client)

    def test_can_import_llm_client_abc(self):
        """Verify LLMClient ABC can be imported."""
        from abc import ABC
        from gaia.llm import LLMClient
        assert issubclass(LLMClient, ABC)

    def test_can_import_not_supported_error(self):
        """Verify NotSupportedError can be imported."""
        from gaia.llm import NotSupportedError
        assert issubclass(NotSupportedError, Exception)

class TestFactory:
    def test_default_creates_lemonade_provider(self):
        """Test factory creates Lemonade by default."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient"):
            client = create_client()
            assert client.provider_name == "Lemonade"

    def test_explicit_provider_selection(self):
        """Test explicit provider parameter."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient"):
            client = create_client(provider="lemonade")
            assert client.provider_name == "Lemonade"

    def test_use_claude_flag(self):
        """Test backward-compatible use_claude flag."""
        with patch("gaia.llm.providers.claude.anthropic"):
            client = create_client(use_claude=True, api_key="test")
            assert client.provider_name == "Claude"

    def test_invalid_provider_raises_error(self):
        """Test unknown provider raises ValueError."""
        with pytest.raises(ValueError, match="Unknown provider"):
            create_client(provider="invalid")

    def test_case_insensitive_provider(self):
        """Test provider names are case-insensitive."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient"):
            client = create_client(provider="LEMONADE")
            assert client.provider_name == "Lemonade"

class TestNotSupportedError:
    def test_error_message_format(self):
        """Test NotSupportedError message format."""
        error = NotSupportedError("TestProvider", "test_method")
        assert "TestProvider" in str(error)
        assert "test_method" in str(error)

    def test_claude_embed_raises_not_supported(self):
        """Test Claude provider raises NotSupportedError for embed."""
        with patch("gaia.llm.providers.claude.anthropic"):
            client = create_client(provider="claude", api_key="test")
        with pytest.raises(NotSupportedError) as exc:
            client.embed(["text"])
        assert "Claude" in str(exc.value)
        assert "embed" in str(exc.value)

class TestProviderMethods:
    def test_lemonade_generate(self):
        """Test LemonadeProvider.generate()."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient") as MockClient:
            mock_backend = Mock()
            mock_backend.chat_completions.return_value = {
                "choices": [{"message": {"content": "Hello"}}]
            }
            MockClient.return_value = mock_backend

            client = create_client()
            response = client.generate("Test")
            assert response == "Hello"

    def test_lemonade_chat(self):
        """Test LemonadeProvider.chat()."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient") as MockClient:
            mock_backend = Mock()
            mock_backend.chat_completions.return_value = {
                "choices": [{"message": {"content": "Hi"}}]
            }
            MockClient.return_value = mock_backend

            client = create_client()
            response = client.chat([{"role": "user", "content": "Hello"}])
            assert response == "Hi"

    def test_streaming_returns_iterator(self):
        """Test streaming returns an iterator."""
        with patch("gaia.llm.providers.lemonade.LemonadeClient") as MockClient:
            mock_backend = Mock()

            def mock_stream():
                yield {"choices": [{"delta": {"content": "Hi"}}]}
                yield {"choices": [{"delta": {"content": " there"}}]}

            mock_backend.chat_completions.return_value = mock_stream()
            MockClient.return_value = mock_backend

            client = create_client()
            result = client.chat([{"role": "user", "content": "Hello"}], stream=True)
            chunks = list(result)
            assert len(chunks) == 2
            assert "".join(chunks) == "Hi there"

File: tests/unit/test_removed_llm_providers.py — retired names and flags are rejected across the factory, SDK, base agent and CLI, before provider import or model initialization.

Regression coverage

test_llm_client_factory.py: supported selection and shared interface.
test_removed_llm_providers.py: rejected legacy names and flags across the factory, SDK, base agent and CLI, before model initialization.
unit/test_lemonade_client_http.py: supported Lemonade operations and their request shapes, against mocked HTTP.
test_lemonade_client.py: the same operations against a live Lemonade Server.
test_lemonade_cloud_models.py: local/cloud lifecycle separation.
test_parallel_tool_calls.py: preservation and execution of native calls.

The unit workflow triggers for changes to shared source and the packaged chat agent. Tests must exercise the retained route as well as migration failures.

Integration Tests

File: tests/integration/test_llm_providers.py

def test_integration_lemonade_generate():
    """Test Lemonade provider with live server."""
    try:
        client = create_client()
        response = client.generate("Say hello")
        assert isinstance(response, str)
        assert len(response) > 0
    except ConnectionError:
        pytest.skip("Lemonade server not running")

def test_integration_lemonade_streaming():
    """Test Lemonade streaming."""
    try:
        client = create_client()
        chunks = list(client.generate("Count to 3", stream=True))
        assert len(chunks) > 0
    except ConnectionError:
        pytest.skip("Lemonade server not running")

def test_integration_lemonade_performance_stats():
    """Test performance stats (Lemonade only)."""
    try:
        client = create_client()
        client.generate("Test")
        stats = client.get_performance_stats()
        assert isinstance(stats, dict)
    except ConnectionError:
        pytest.skip("Lemonade server not running")

Dependencies

Required Packages

# pyproject.toml
[project]
dependencies = [
    "openai>=1.0.0",      # OpenAI-compatible HTTP client used by LemonadeClient
    "httpx>=0.24.0",      # HTTP client with timeout support
    "requests>=2.31.0",   # For performance stats/control endpoints
    "python-dotenv>=1.0.0",  # Environment variable management
]

[project.optional-dependencies]
claude = ["anthropic>=0.18.0"]  # Claude API support

Import Dependencies

Factory (factory.py):

import importlib
from typing import Optional
from .base_client import LLMClient

Base Client (base_client.py):

from abc import ABC, abstractmethod
from typing import Iterator, Union
from .exceptions import NotSupportedError

LemonadeProvider (providers/lemonade.py):

from typing import Iterator, Optional, Union
from ..base_client import LLMClient
from ..lemonade_client import LemonadeClient, DEFAULT_MODEL_NAME
# DEFAULT_MODEL_NAME = "Gemma-4-E4B-it-GGUF"

ClaudeProvider (providers/claude.py):

from typing import Iterator, Optional, Union
try:
    import anthropic  # Requires: pip install anthropic
except ImportError:
    anthropic = None
from ..base_client import LLMClient

Usage Examples

Example 1: Basic Usage with Factory

from gaia.llm import create_client

# Default Lemonade provider
client = create_client()

# Non-streaming generation
response = client.generate("Write a hello world program in Python")
print(response)

# Get performance stats
stats = client.get_performance_stats()
print(f"Speed: {stats.get('tokens_per_second', 'N/A')} tokens/sec")

Example 2: Explicit Provider Selection

from gaia.llm import create_client

# Lemonade (local)
lemonade = create_client(provider="lemonade", model="Qwen3-0.6B-GGUF")

# Claude
claude = create_client(provider="claude", api_key="sk-ant-...")

# Backward-compatible flag
legacy_claude = create_client(use_claude=True, api_key="sk-ant-...")

Example 3: Handling Unsupported Features

from gaia.llm import create_client, NotSupportedError

# Create Claude client
client = create_client(provider="claude", api_key="sk-ant-...")

# This works (Claude supports vision)
description = client.vision([image_bytes], "Describe this image")

# This raises NotSupportedError (Claude doesn't support embeddings)
try:
    embeddings = client.embed(["Hello world"])
except NotSupportedError as e:
    print(f"Feature not available: {e}")
    # Output: "Claude does not support embed"

Example 4: Chat with Message History

from gaia.llm import create_client

client = create_client()

# Build conversation history
messages = [
    {"role": "user", "content": "What's 2+2?"},
    {"role": "assistant", "content": "2+2 equals 4."},
    {"role": "user", "content": "What about 3+3?"}
]

# Use chat() method
response = client.chat(messages)
print(response)  # "3+3 equals 6."

Example 5: Streaming Responses

from gaia.llm import create_client

client = create_client()

# Streaming with generate()
print("AI: ", end="", flush=True)
for chunk in client.generate("Tell me a short story", stream=True):
    print(chunk, end="", flush=True)
print()

# Streaming with chat()
for chunk in client.chat([{"role": "user", "content": "Hello"}], stream=True):
    print(chunk, end="", flush=True)

Example 6: Embeddings (Lemonade only)

from gaia.llm import create_client

lemonade = create_client()
embeddings = lemonade.embed(["Hello world", "How are you?"])
print(f"Embedding dimensions: {len(embeddings[0])}")

Example 7: Vision (Lemonade and Claude only)

from gaia.llm import create_client

# With Claude
claude = create_client(provider="claude", api_key="sk-ant-...")
with open("image.jpg", "rb") as f:
    image_data = f.read()
description = claude.vision([image_data], "Describe what you see")
print(description)

Example 8: Model Management (Lemonade only)

from gaia.llm import create_client

client = create_client()

# Load a specific model
client.load_model("Qwen3-0.6B-GGUF")

# Generate
response = client.generate("Hello")

# Get performance stats
stats = client.get_performance_stats()
print(f"Speed: {stats.get('tokens_per_second', 'N/A')} tokens/sec")

# Unload model
client.unload_model()

Example 9: Remote Lemonade Server

from gaia.llm import create_client

# Connect to remote server
client = create_client(base_url="http://192.168.1.100:13305")

response = client.generate("Hello from remote server")
print(response)

Example 10: System Prompts

from gaia.llm import create_client

# Set default system prompt
client = create_client(
    system_prompt="You are a helpful coding assistant."
)

# System prompt automatically prepended to chat messages
response = client.chat([
    {"role": "user", "content": "Write a binary search function"}
])
print(response)

Third-Party LLM Integration

The Lemonade provider sends inference over OpenAI-compatible endpoints, but before a local-model request it also calls Lemonade’s model-lifecycle API (for example /load). A generic OpenAI-compatible server is therefore not a drop-in replacement: configure the third-party or on-prem model in a Lemonade server and point GAIA at that server. See gateway migration.

Inference Endpoints

GAIA sends inference requests in the OpenAI-compatible shapes below.

Completions Endpoint

Default: POST /v1/completionsUsed for pre-formatted prompts

Chat Completions Endpoint

POST /v1/chat/completionsUsed for structured conversations with message history

Completions Endpoint

Request
Response
Streaming

{
  "model": "your-model-name",
  "prompt": "Your prompt text here",
  "stream": false,
  "temperature": 0.1,
  "max_tokens": 2048
}

{
  "id": "cmpl-abc123",
  "object": "text_completion",
  "created": 1677652288,
  "model": "your-model-name",
  "choices": [
    {
      "text": "Generated response text",
      "index": 0,
      "finish_reason": "stop"
    }
  ]
}

Server-Sent Events (SSE) format:

data: {"choices": [{"text": "chunk1", "index": 0}]}

data: {"choices": [{"text": "chunk2", "index": 0}]}

data: [DONE]

Chat Completions Endpoint

Request
Response
Streaming

{
  "model": "your-model-name",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello!"}
  ],
  "stream": false,
  "temperature": 0.1
}

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "your-model-name",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hi! How can I help you?"
      },
      "finish_reason": "stop"
    }
  ]
}

Server-Sent Events (SSE) with delta chunks:

data: {"choices": [{"delta": {"content": "Hi!"}, "index": 0}]}

data: {"choices": [{"delta": {"content": " How"}, "index": 0}]}

data: [DONE]

Configuration

Point GAIA at the Lemonade server that fronts your model:

Environment Variable
Direct Initialization

Linux

export LEMONADE_BASE_URL="http://your-lemonade-server:8080"

Windows (PowerShell)

$env:LEMONADE_BASE_URL="http://your-lemonade-server:8080"

Windows (CMD)

set LEMONADE_BASE_URL=http://your-lemonade-server:8080

from gaia.llm import create_client

client = create_client(base_url="http://your-lemonade-server:8080")
response = client.generate("Hello world")

URL Normalization: LemonadeClient automatically appends /api/v1 if not present:

http://localhost:8080 → http://localhost:8080/api/v1
If your server uses /v1 instead, provide the full path: http://localhost:8080/v1

Example Integration

from gaia.llm import create_client

# Connect to the Lemonade server that fronts your model
client = create_client(
    base_url="http://your-lemonade-server:8080",
    model="your-catalog-model-id",
)

# Test connection
response = client.generate("Hello, are you working?")
print(response)

from gaia.llm import create_client

client = create_client(
    base_url="http://your-lemonade-server:8080",
    model="your-catalog-model-id",
)

# Stream response chunks
for chunk in client.generate("Tell me a story", stream=True):
    print(chunk, end="", flush=True)

Troubleshooting

Connection Errors

Problem: Requests fail because the server can’t be reached (connection refused, DNS, or TLS)Solutions:

Verify the server is running:

curl http://your-lemonade-server:port/api/v1/models

Check firewall settings
Ensure correct base URL format

Test with explicit base URL:

client = create_client(base_url="http://localhost:8080/api/v1")

404 Endpoint Errors

Problem: Requests fail with HTTP 404Solutions:

Verify API path structure: /v1 vs /api/v1
Confirm base_url points at the Lemonade server, not the raw upstream endpoint

Use chat method explicitly if needed:

client.chat([{"role": "user", "content": "Test"}])

Model Not Found

Problem: Model errors or “model not loaded”Solutions:

Specify model explicitly:

client.generate("Test", model="your-model-name")

List the models the server exposes:

curl http://your-lemonade-server:port/api/v1/models

Use the exact model ID from that catalog

Streaming Issues

Problem: Streaming responses not workingSolutions:

Verify the server supports Server-Sent Events (SSE)
Check Content-Type headers: text/event-stream

Test non-streaming first:

response = client.generate("Test", stream=False)

Enable debug logging:

import logging
logging.basicConfig(level=logging.DEBUG)

Documentation Updates Required

docs/sdk/sdks/llm.mdx

Add to LLM Section:

### LLM Client

**Import:** `from gaia.llm import create_client, LLMClient, NotSupportedError`

**Purpose:** Provider-based LLM client with factory pattern for local and cloud backends.

**Features:**
- Factory-based client creation with `create_client()`
- Two providers: Lemonade (local), Claude
- Abstract base class for type safety
- `NotSupportedError` for unsupported features
- Streaming and non-streaming generation
- Backward-compatible flags

**Quick Start:**
```python
from gaia.llm import create_client

# Local LLM (default)
client = create_client()
response = client.generate("Hello world")

# Streaming
for chunk in client.generate("Tell me a story", stream=True):
    print(chunk, end="")

# Claude API
claude = create_client(provider="claude", api_key="sk-ant-...")
response = claude.generate("Explain Python decorators")

# Backward-compatible
client = create_client(use_claude=True)

LLMClient Technical Specification

​Overview

​Removed adapters and migration

​Requirements

​Functional Requirements

​Non-Functional Requirements

​API Specification

​Package Structure

​Package Exports (__init__.py)

​Factory Function (factory.py)

​Abstract Base Class (base_client.py)

​NotSupportedError (exceptions.py)

​Provider Implementations

​LemonadeProvider (providers/lemonade.py)

​ClaudeProvider (providers/claude.py)

​Implementation Details

​Provider Selection Logic

​Lazy Provider Loading

​NotSupportedError Pattern

​Sampling Defaults

​Provider-Specific Implementation

​Native tool-call transport

​Error boundaries

​Testing Requirements

​Unit Tests

​Regression coverage

​Integration Tests

​Dependencies

​Required Packages

​Import Dependencies

​Usage Examples

​Example 1: Basic Usage with Factory

​Example 2: Explicit Provider Selection

​Example 3: Handling Unsupported Features

​Example 4: Chat with Message History

​Example 5: Streaming Responses

​Example 6: Embeddings (Lemonade only)

​Example 7: Vision (Lemonade and Claude only)

​Example 8: Model Management (Lemonade only)

​Example 9: Remote Lemonade Server

​Example 10: System Prompts

​Third-Party LLM Integration

​Inference Endpoints

Completions Endpoint

Chat Completions Endpoint

​Completions Endpoint

​Chat Completions Endpoint

​Configuration

​Example Integration

​Troubleshooting

​Documentation Updates Required

​docs/sdk/sdks/llm.mdx

Overview

Removed adapters and migration

Requirements

Functional Requirements

Non-Functional Requirements

API Specification

Package Structure

Package Exports (`init.py`)

Factory Function (`factory.py`)

Abstract Base Class (`base_client.py`)

NotSupportedError (`exceptions.py`)

Provider Implementations

LemonadeProvider (`providers/lemonade.py`)

ClaudeProvider (`providers/claude.py`)

Implementation Details

Provider Selection Logic

Lazy Provider Loading

NotSupportedError Pattern

Sampling Defaults

Provider-Specific Implementation

Native tool-call transport

Error boundaries

Testing Requirements

Unit Tests

Regression coverage

Integration Tests

Dependencies

Required Packages

Import Dependencies

Usage Examples

Example 1: Basic Usage with Factory

Example 2: Explicit Provider Selection

Example 3: Handling Unsupported Features

Example 4: Chat with Message History

Example 5: Streaming Responses

Example 6: Embeddings (Lemonade only)

Example 7: Vision (Lemonade and Claude only)

Example 8: Model Management (Lemonade only)

Example 9: Remote Lemonade Server

Example 10: System Prompts

Third-Party LLM Integration

Inference Endpoints

Completions Endpoint

Chat Completions Endpoint

Configuration

Example Integration

Troubleshooting

Documentation Updates Required

docs/sdk/sdks/llm.mdx