Voice Interaction

Source Code: src/gaia/talk/ · src/gaia/audio/

GAIA’s talk mode enables natural voice-based interaction with LLMs using Whisper for automatic speech recognition (ASR) and Kokoro for text-to-speech (TTS). Have natural conversations with AI through your microphone and speakers. Both models run inside Lemonade Server, the same server that runs the chat model — GAIA itself installs no speech model runtime. The first gaia talk downloads the Whisper model (~150 MB for base) and the Kokoro voice model (~350 MB) through Lemonade.

First time here? Complete the Setup guide first to install GAIA and its dependencies.

Quick Start

System Prerequisites

Talk mode needs a running Lemonade Server (gaia init installs it) and PortAudio for the microphone and speakers.

Linux
Windows / macOS

sudo apt-get install -y libportaudio2

PortAudio ships inside the sounddevice wheel, so there is nothing extra to install.

Transcribing an existing recording is no longer part of talk mode — point the GAIA agent at the file instead. See Transcription & Meeting Notes.

Install talk extras

With GAIA installed, add the talk extras:

uv pip install "amd-gaia[talk]"

Launch Talk Mode

Start voice conversation:

Full Voice (ASR + TTS)
ASR Only

gaia talk

gaia talk --no-tts

Start Speaking

When you see ⠴ Listening..., you can start talking

[2025-02-06 11:56:30] | INFO | Starting audio processing thread...
[2025-02-06 11:56:30] | INFO | Listening for voice input...
⠴ Listening...

Say “stop” to end the session, or press Ctrl+C.

GAIA mutes the microphone while it is speaking, so it never answers its own reply. Wait for it to finish before you speak.

Voice Commands

Exit Session

Say “stop” (or press Ctrl+C)

Clear History

Say “restart”

Trigger Response

Pause longer than --silence-threshold (default 0.5 seconds)

Stop Playback

Press Enter while GAIA is speaking (terminal only)

Configuration Options

The microphone stays paused throughout playback. If the output device stalls beyond the synthesized audio duration plus five seconds, voice chat reports an error and keeps capture paused. Check the output device, then restart voice chat; gaia talk --no-tts disables spoken output. Customize your voice interaction experience:

Whisper Model Size
Audio Device
Mic Sensitivity
Model & Backend
Performance Stats

# Choose model size for speech recognition
gaia talk --whisper-model-size medium

# Available: tiny, base, small, medium, large, turbo

Each size maps to a Lemonade Whisper model (base is Whisper-Base, large is Whisper-Large-v3, turbo is Whisper-Large-v3-Turbo). Larger models are more accurate but slower and a bigger first download.

# Specify which microphone to use
gaia talk --audio-device-index 2

Use gaia test --test-type asr-list-audio-devices to list available devices

# Adjust microphone sensitivity for voice detection
gaia talk --mic-threshold 0.001

# Lower values = more sensitive (picks up quieter speech)
# Higher values = less sensitive (needs louder speech)
# Default: 0.003

If voice detection is not picking up your speech, try lowering the threshold. If it triggers on background noise, raise it.

# Use a specific local model and a longer reply budget
gaia talk --model Qwen3-Coder-30B-A3B-Instruct-GGUF --max-tokens 1024

# Point at a Lemonade Server on another host or port
gaia talk --base-url http://192.168.1.20:13305/api/v1

# Use a cloud provider instead of local Lemonade
gaia talk --use-claude                            # claude-sonnet-5 by default
gaia talk --use-claude --claude-model claude-opus-5

Defaults: Gemma-4-E4B-it-GGUF, 512 tokens, local Lemonade. --use-claude needs ANTHROPIC_API_KEY. --use-chatgpt was removed — configure the model in Lemonade and point --base-url at it.

# Print performance statistics after each reply (--show-stats also works)
gaia talk --stats

Document Q&A with Voice

Voice interaction supports document-based Q&A through RAG (Retrieval-Augmented Generation). Ask questions about your PDF documents using natural speech!

Quick Start with Documents

# Voice chat with a document
gaia talk --index manual.pdf

# Or use short form
gaia talk -i guide.pdf

# Without text-to-speech
gaia talk --index manual.pdf --no-tts

Use Cases

Technical Support

Voice chat with product manuals and troubleshooting guides

Research

Speak questions about research papers and documentation

Learning

Voice interaction with textbooks and educational materials

Accessibility

Hands-free document Q&A for users with mobility needs

Field Work

Voice queries about procedures when hands are busy

Documentation

Quick reference lookup while working

How It Works

Document Indexing

PDFs are automatically indexed when you start talk with --index

Voice Input

Speak your question about the documents

Context Retrieval

Relevant document sections are retrieved automatically

Voice Response

AI answers based on document context and speaks the response

See the Chat documentation - Document Q&A section for more details on RAG capabilities

Testing ASR Components

Talk mode’s ASR covers live microphone input only. To transcribe a file you already have, see Transcription & Meeting Notes.

List Audio Devices

Discover available audio input devices:

gaia test --test-type asr-list-audio-devices

Microphone Recording Test

Test real-time transcription from your microphone:

gaia test --test-type asr-microphone --recording-duration 15

Options:

--recording-duration: Recording duration in seconds (default: 10)
--whisper-model-size: Whisper size Lemonade transcribes with (default: “base”)
--audio-device-index: Specific microphone (optional)

Testing TTS Components

Test text-to-speech capabilities with various test modes:

Text Preprocessing

Test how TTS processes and formats text:

gaia test --test-type tts-preprocessing

gaia test --test-type tts-preprocessing --test-text "Your test text here"

Streaming Playback

Test real-time audio generation and playback:

gaia test --test-type tts-streaming --test-text "Your test text here"

Test Output Includes

Processing progress
Playback progress
Currently spoken text
Performance metrics

Audio File Generation

Generate and save audio to WAV file:

gaia test --test-type tts-audio-file \
  --test-text "Your test text here" \
  --output-audio-file ./test_output.wav

Troubleshooting

Audio Device Errors

Try different --audio-device-index values
List available devices: gaia test --test-type asr-list-audio-devices
Check system audio input settings (Settings > Audio > Input)
Ensure correct microphone is selected as default input device

Poor ASR Accuracy

Try larger Whisper models: --whisper-model-size medium or large
Ensure you’re in a quiet environment
Speak clearly at a moderate pace
Check microphone positioning and quality
Verify microphone is not muted

No Voice Response (TTS)

Check system audio output/speaker settings
Verify TTS is enabled (not using --no-tts flag)
Ensure system volume is not muted
Check that Lemonade Server is running and lists kokoro-v1
Test with: gaia test --test-type tts-streaming

Speech Recognition Failed

gaia talk stops with Speech recognition failed: … when Lemonade cannot transcribe an utterance — usually because the server stopped. The error says how to start it on your system; then run gaia talk again. Talk mode never falls back to a local model.

Voice Input Not Recognized

Check microphone permissions
Verify microphone is working in other applications
Test with: gaia test --test-type asr-microphone
Adjust --audio-device-index if multiple microphones

RAG Issues

Missing RAG dependencies:

uv pip install -e ".[rag]"

Other issues:

PDF processing errors: Ensure PDFs have extractable text (not scanned images)
Slow indexing: Use --stats to monitor; larger documents take time
Context not used: Verify documents indexed successfully at startup
Empty responses: Check PDFs contain extractable text

Best Practices

Optimal Environment

Use in quiet environments for best recognition accuracy

Speech Clarity

Speak clearly and at moderate pace

Model Selection

Balance accuracy vs. performance based on your hardware

Natural Pauses

Use natural pauses to trigger AI responses

Next Steps

Document Q&A

Chat with documents using RAG-powered retrieval

CLI Reference

Explore all command-line options

Development Guide

Build custom voice-enabled agents

Features Overview

Discover all GAIA capabilities

Getting Started

C++ Framework

Python Framework

Voice Interaction

Quick Start

System Prerequisites

Voice Commands

Exit Session

Clear History

Trigger Response

Stop Playback

Configuration Options

Document Q&A with Voice

Quick Start with Documents

Use Cases

Technical Support

Research

Learning

Accessibility

Field Work

Documentation

How It Works

Testing ASR Components

List Audio Devices

Microphone Recording Test

Testing TTS Components

Text Preprocessing

Streaming Playback

Audio File Generation

Troubleshooting

Best Practices

Optimal Environment

Speech Clarity

Model Selection

Natural Pauses

Next Steps

Document Q&A

CLI Reference

Development Guide

Features Overview

​Quick Start

​System Prerequisites

​Voice Commands

Exit Session

Clear History

Trigger Response

Stop Playback

​Configuration Options

​Document Q&A with Voice

​Quick Start with Documents

​Use Cases

Technical Support

Research

Learning

Accessibility

Field Work

Documentation

​How It Works

​Testing ASR Components

​List Audio Devices

​Microphone Recording Test

​Testing TTS Components

​Text Preprocessing

​Streaming Playback

​Audio File Generation

​Troubleshooting

​Best Practices

Optimal Environment

Speech Clarity

Model Selection

Natural Pauses

​Next Steps

Document Q&A

CLI Reference

Development Guide

Features Overview

Quick Start

System Prerequisites

Voice Commands

Configuration Options

Document Q&A with Voice

Quick Start with Documents

Use Cases

How It Works

Testing ASR Components

List Audio Devices

Microphone Recording Test

Testing TTS Components

Text Preprocessing

Streaming Playback

Audio File Generation

Troubleshooting

Best Practices

Next Steps