Medical Intake Agent

Source Code: hub/agents/python/emr/ · hub/agents/python/emr/gaia_agent_emr/dashboard/

Demonstration Application: This is a proof-of-concept demo showcasing AMD Ryzen AI capabilities. Not intended for production use with real patient data. Do not use with actual PHI (Protected Health Information).

The GAIA Medical Intake Agent demonstrates automated patient intake form processing using Vision Language Models (VLM). Drop a scanned intake form into the watch folder, and within seconds the agent extracts patient demographics, insurance information, medical history, and more—storing everything in a searchable SQLite database. All processing happens 100% locally on AMD Ryzen AI hardware. No cloud APIs, no data leaving your machine—critical for healthcare scenarios where patient privacy is paramount. The agent includes a real-time dashboard for monitoring processing status, viewing patient records, and querying the database using natural language.

Want to learn how it works? See the EMR Agent Playbook for a step-by-step guide to building this agent from scratch.

How It Works

The EMR agent combines three AI models in a sophisticated pipeline that runs entirely on your local hardware:

Vision Language Model (VLM) - The Gemma-4-E4B-it model “sees” the intake form image and extracts text using a carefully crafted prompt that guides it to identify specific fields (name, DOB, allergies, medications, etc.). Unlike traditional OCR, the VLM understands context—it knows that “DOB” means date of birth and can handle handwritten entries, checkboxes, and varied form layouts.
LLM Validation & Querying - The Qwen3.5-35B-A3B-GGUF model (a Mixture-of-Experts architecture that activates only 3B parameters per inference) validates extracted data, handles natural language queries, and generates SQL to search the patient database. When you ask “Which patients have penicillin allergies?”, the LLM translates this to proper SQL.
Embedding Model - The user.embeddinggemma-300m-GGUF (EmbeddingGemma 300M) model creates vector embeddings for semantic similarity search, enabling fuzzy matching when looking up returning patients or finding related records.

Why Local Matters: Running on-device with AMD Ryzen AI means sub-second inference latency, no per-request API costs, and complete data sovereignty. A typical intake form processes in 10-15 seconds on AMD Ryzen AI MAX+ hardware.

Key Features

Automatic file watching - Monitors a directory for new intake forms
Drag-and-drop upload - Drop files directly into the Watch Folder panel
VLM-powered extraction - Uses Gemma-4-E4B-it for OCR and data extraction
Local database storage - SQLite with full patient record schema
New/returning detection - Identifies returning patients and flags changes
Critical alerts - Automatic alerts for allergies and missing fields
Web dashboard - Real-time monitoring with SSE updates
Cumulative efficiency metrics - Track total time saved across all processed forms

Required Models

The EMR agent uses three models, downloaded automatically on first run via gaia-emr init:

Model	Size	Purpose
Qwen3.5-35B-A3B-GGUF	18.6 GB	LLM for chat queries and patient search
Gemma-4-E4B-it-GGUF	~3 GB	Vision language model for form extraction
user.embeddinggemma-300m-GGUF	~334 MB	Embedding model for similarity search

Disk Space: Ensure you have at least 25 GB of free disk space for model downloads.

Prerequisites

Complete the Setup guide first to install Lemonade Server and uv. Then install GAIA with the EMR extras in a Python 3.12 virtual environment.

Windows
Linux
From Source (Dev)

uv venv .venv --python 3.12
.\.venv\Scripts\Activate.ps1
uv pip install "amd-gaia[api,rag]"
gaia-emr --help

If activation fails with a script execution error, run once: Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser, then retry.

uv venv .venv --python 3.12
source .venv/bin/activate
uv pip install "amd-gaia[api,rag]"
gaia-emr --help

For contributors. Clone the repo and install in editable mode (changes take effect without reinstalling):

git clone https://github.com/amd/gaia.git
cd gaia
uv venv .venv --python 3.12
source .venv/bin/activate        # Windows: .\.venv\Scripts\Activate.ps1
uv pip install -e ".[dev,api,rag]"
gaia-emr --help

The api extra provides FastAPI and uvicorn for the web dashboard; the rag extra provides PyMuPDF for PDF processing. uv downloads Python 3.12 automatically if it isn’t already installed.

Having issues? Check the Troubleshooting guide, create an issue on GitHub, or contact us at [email protected].

Quick Start

Step 1: Initialize (First Time Only)

Download and load all required models before first use:

gaia-emr init

This command:

Checks Lemonade server is running and context size is configured
Downloads and loads all required models:
- VLM: Gemma-4-E4B-it-GGUF (form extraction)
- LLM: Qwen3.5-35B-A3B-GGUF (chat/query processing)
- Embedding: user.embeddinggemma-300m-GGUF (similarity search)
Verifies all models are loaded and ready

Context Size: For best results, set Lemonade context size to 32768. Right-click the Lemonade tray icon → Settings → Context Size → 32768.

Partial Success: If the LLM fails to download but VLM succeeds, form extraction will still work. Chat queries and natural language patient search require the LLM. Run gaia-emr init again to retry failed downloads.

Step 2: Launch

CLI App
Desktop App

Download sample forms

Download sample intake forms from GitHub and save them to a local directory (e.g., ./intake-forms/).

Dev Install: See the EMR agent source at hub/agents/python/emr/ for sample forms and configuration.

Start watching for forms

gaia-emr watch --watch-dir ./intake-forms

The agent will process the sample forms and display extracted patient data.

Example Output

+-----------------------------------+
| Medical Intake Agent              |
| Automatic Patient Form Processing |
+-----------------------------------+
  Watch folder: ./intake-forms
  Database:     ./data/patients.db

  File            Size      Hash          Status
  --------------  --------  ------------  ------
  IMG_2992.jpg    1.7 MB    eaabe23e...   new
  IMG_2993.jpg    887.6 KB  03f2391e...   new
  IMG_2995.jpg    2.3 MB    348071a9...   new
  IMG_2996.jpg    2.1 MB    7a73ea84...   new

Processing Pipeline (7 steps)
  File: IMG_2992.jpg
  [5/7] Extracting patient data
  VLM extracting from image...
  Extracted 1844 chars in 14.12s
Pipeline complete in 14.3s -> Alice Williams

Extracted Fields
  Identity
    first_name           Alice
    last_name            Williams
    date_of_birth        1980-04-04
  Contact
    phone                (411) 413-1234
    email                [email protected]
  Insurance
    insurance_provider   Medicaid Demo
  ...

34 fields extracted

Query patients

Once processing completes, you can query the database using natural language. The agent uses tool calling to translate your questions into SQL queries and retrieve results from the SQLite database.

Which patients have allergies?

Show me all patients processed today

Summarize today's intake forms

Type quit or press Ctrl+C to stop.

CLI Commands Reference

gaia-emr init

gaia-emr watch --watch-dir ./forms

gaia-emr dashboard

gaia-emr query "patients with allergies"

gaia-emr reset

gaia-emr -h

Command	Description
`init`	Download all required models (VLM, LLM, embedding)
`watch`	Watch folder and process forms
`process`	Process a single form file and exit
`dashboard`	Launch web dashboard (Electron or browser)
`query`	One-shot database query
`stats`	Print database statistics (patient count, file counts, etc.)
`reset`	Delete database and start fresh
`test`	Run end-to-end self-test against bundled samples
`-h`	Full command reference

The recommended way to use the EMR Dashboard is via the native Electron desktop app.

Launch the desktop app

gaia-emr dashboard

The dashboard opens automatically in a native desktop window.

First run will install Electron dependencies automatically. The default database is ./data/patients.db.

Add intake forms

Drag and drop intake form images or PDFs directly onto the Watch Folder panel in the dashboard. The agent will process them automatically.

Download sample intake forms from GitHub to test the agent.

Dev Install: Build the frontend (first time only)

If you installed from source (Windows/Linux Dev tabs), you need to build the frontend first:

cd hub/agents/python/emr/gaia_agent_emr/dashboard/frontend

npm install

npm run build

Then return to the repository root:

cd ../../../../../..

Dashboard Features

The dashboard includes four main views:

Dashboard - Real-time stats, cumulative efficiency metrics, and live processing feed
Patient Database - Searchable patient list with detailed records
Chat - Natural language queries about patients
Settings - Configure watch directory, upload files, and manage database

Watch Folder Panel: The left column displays a Watch Folder panel with status indicators:

Green dot - Processed files
Red flashing dot - Currently processing
Orange dot - Queued for processing

Drag-and-Drop: Drop intake form images or PDFs directly onto the Watch Folder panel to upload and process them instantly.

Command Options

gaia-emr dashboard

gaia-emr dashboard --watch-dir /path/to/forms

gaia-emr dashboard --browser

gaia-emr dashboard --port 3000

gaia-emr dashboard --no-open

Option	Default	Description
`--watch-dir`	`./intake_forms`	Directory to monitor for intake forms
`--db`	`./data/patients.db`	SQLite database path
`--port`	`8080`	Server port
`--browser`	off	Open in browser instead of Electron
`--no-open`	off	Don’t auto-open, server only

If Electron/Node.js is not available, the dashboard automatically falls back to opening in your default web browser.

Supported Intake Form Formats

The agent accepts scanned or photographed intake forms in these formats:

Extension	Processing
`.png`, `.jpg`, `.jpeg`	Direct image processing
`.pdf`	Converted to image via PyMuPDF
`.tiff`, `.bmp`	Direct image processing

Under the Hood

Image preprocessing — Before reaching the VLM, forms are auto-rotated from EXIF orientation (critical for phone photos), scaled to a max 1024px dimension, and JPEG-compressed (quality 85) to balance extraction quality against image token count.
Returning-patient detection — A multi-signal lookup combines exact name + DOB match, fuzzy matching for misspellings (“Jon Smith” → “John Smith”), and user.embeddinggemma-300m-GGUF vector similarity. When a returning patient is detected, changes from their previous record (new allergies, updated insurance) are highlighted.
Real-time dashboard — The FastAPI backend streams processing events to the React frontend over Server-Sent Events (SSE), so the UI updates within ~100ms of file detection without polling or WebSockets.
Safe DB queries — Natural language questions are answered via LLM tool calling: the model never writes SQL directly, it calls predefined, validated tools (e.g. search_patients) that construct queries safely.

Troubleshooting

Context Size Too Small

Context size too small! Image requires 4203 tokens but model context is only 4096.

Large form images can require 4,000–8,000+ tokens. Set Lemonade context size to 32768: right-click the Lemonade tray icon → Settings → Context Size → 32768, then restart the model.

Model Download Failed / Init Fails

Download succeeded but failed to rename file: The process cannot access the file

Run gaia-emr init again to resume. If it keeps failing, close any apps using the model files, delete the partial/corrupted model folder from Lemonade’s cache (Windows: %LOCALAPPDATA%\AMD\LemonadeModels\, Linux: ~/.local/share/lemonade/models/), restart Lemonade Server, then re-run gaia-emr init.

PyMuPDF Required

ERROR: PyMuPDF required for PDF processing

Install the RAG extra: pip install "amd-gaia[rag]".

JSON Parse Failed

WARNING: Failed to parse extraction for: form.jpg

The VLM output wasn’t valid JSON — usually a low-quality or unclear form image. Check image clarity, or re-run gaia-emr test <file> to inspect the raw extraction.

Database Locked

ERROR: database is locked

Only one agent process should access a given database file at a time. Stop other gaia-emr processes pointed at the same --db path.

Learn More

Part 1: Getting Started

Build this agent from scratch and understand the core components

Part 2: Dashboard & API

Deep dive into the web dashboard and REST API endpoints

Part 3: Architecture

Database schema, processing pipeline, and system design

Getting Started

C++ Framework

Python Framework

Medical Intake Agent

How It Works

Key Features

Required Models

Prerequisites

Quick Start

Step 1: Initialize (First Time Only)

Step 2: Launch

Download sample forms

Start watching for forms

Query patients

CLI Commands Reference

Launch the desktop app

Add intake forms

Dashboard Features

Command Options

Supported Intake Form Formats

Under the Hood

Troubleshooting

Context Size Too Small

Model Download Failed / Init Fails

PyMuPDF Required

JSON Parse Failed

Database Locked

Learn More

Part 1: Getting Started

Part 2: Dashboard & API

Part 3: Architecture

​How It Works

​Key Features

​Required Models

​Prerequisites

​Quick Start

​Step 1: Initialize (First Time Only)

​Step 2: Launch

​Download sample forms

​Start watching for forms

​Query patients

​CLI Commands Reference

​Launch the desktop app

​Add intake forms

​Dashboard Features

​Command Options

​Supported Intake Form Formats

​Under the Hood

​Troubleshooting

​Context Size Too Small

​Model Download Failed / Init Fails

​PyMuPDF Required

​JSON Parse Failed

​Database Locked

​Learn More

Part 1: Getting Started

Part 2: Dashboard & API

Part 3: Architecture

How It Works

Key Features

Required Models

Prerequisites

Quick Start

Step 1: Initialize (First Time Only)

Step 2: Launch

Download sample forms

Start watching for forms

Query patients

CLI Commands Reference

Launch the desktop app

Add intake forms

Dashboard Features

Command Options

Supported Intake Form Formats

Under the Hood

Troubleshooting

Context Size Too Small

Model Download Failed / Init Fails

PyMuPDF Required

JSON Parse Failed

Database Locked

Learn More