> ## Documentation Index > Fetch the complete documentation index at: https://amd-gaia.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Medical Intake Agent > Automate patient intake form processing with VLM extraction and local database storage **Source Code:** [`hub/agents/python/emr/`](https://github.com/amd/gaia/tree/main/hub/agents/python/emr) · [`hub/agents/python/emr/gaia_agent_emr/dashboard/`](https://github.com/amd/gaia/tree/main/hub/agents/python/emr/gaia_agent_emr/dashboard) **Demonstration Application:** This is a proof-of-concept demo showcasing AMD Ryzen AI capabilities. Not intended for production use with real patient data. Do not use with actual PHI (Protected Health Information). The GAIA Medical Intake Agent demonstrates automated patient intake form processing using Vision Language Models (VLM). Drop a scanned intake form into the watch folder, and within seconds the agent extracts patient demographics, insurance information, medical history, and more—storing everything in a searchable SQLite database. All processing happens **100% locally** on AMD Ryzen AI hardware. No cloud APIs, no data leaving your machine—critical for healthcare scenarios where patient privacy is paramount. The agent includes a real-time dashboard for monitoring processing status, viewing patient records, and querying the database using natural language. **Want to learn how it works?** See the [EMR Agent Playbook](/docs/playbooks/emr-agent/part-1-getting-started) for a step-by-step guide to building this agent from scratch. ## How It Works The EMR agent combines three AI models in a sophisticated pipeline that runs entirely on your local hardware: ```mermaid theme={null} %%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E2A33E', 'primaryTextColor':'#1a1a1a', 'primaryBorderColor':'#A87B2D', 'lineColor':'#A87B2D', 'edgeLabelBackground':'#ffffff', 'fontFamily':'system-ui, -apple-system, sans-serif'}}}%% flowchart TD A[/"Intake Form"/] --> B(["VLM Extraction"]) B --> C(["LLM Validation"]) C --> D[("SQLite Database")] D --> E[/"Dashboard & Queries"/] style A fill:#f8f9fa,stroke:#dee2e6,stroke-width:2px,color:#495057 style B fill:#E2A33E,stroke:#A87B2D,stroke-width:2px,color:#1a1a1a style C fill:#EFC480,stroke:#E2A33E,stroke-width:2px,color:#1a1a1a style D fill:#6c757d,stroke:#495057,stroke-width:2px,color:#fff style E fill:#28a745,stroke:#1e7e34,stroke-width:2px,color:#fff linkStyle 0,1,2,3 stroke:#E2A33E,stroke-width:2px ``` 1. **Vision Language Model (VLM)** - The Gemma-4-E4B-it model "sees" the intake form image and extracts text using a carefully crafted prompt that guides it to identify specific fields (name, DOB, allergies, medications, etc.). Unlike traditional OCR, the VLM understands context—it knows that "DOB" means date of birth and can handle handwritten entries, checkboxes, and varied form layouts. 2. **LLM Validation & Querying** - The Qwen3.5-35B-A3B-GGUF model (a Mixture-of-Experts architecture that activates only 3B parameters per inference) validates extracted data, handles natural language queries, and generates SQL to search the patient database. When you ask "Which patients have penicillin allergies?", the LLM translates this to proper SQL. 3. **Embedding Model** - The `user.embeddinggemma-300m-GGUF` (EmbeddingGemma 300M) model creates vector embeddings for semantic similarity search, enabling fuzzy matching when looking up returning patients or finding related records. **Why Local Matters:** Running on-device with AMD Ryzen AI means sub-second inference latency, no per-request API costs, and complete data sovereignty. A typical intake form processes in 10-15 seconds on AMD Ryzen AI MAX+ hardware. ## Key Features * **Automatic file watching** - Monitors a directory for new intake forms * **Drag-and-drop upload** - Drop files directly into the Watch Folder panel * **VLM-powered extraction** - Uses Gemma-4-E4B-it for OCR and data extraction * **Local database storage** - SQLite with full patient record schema * **New/returning detection** - Identifies returning patients and flags changes * **Critical alerts** - Automatic alerts for allergies and missing fields * **Web dashboard** - Real-time monitoring with SSE updates * **Cumulative efficiency metrics** - Track total time saved across all processed forms ## Required Models The EMR agent uses three models, downloaded automatically on first run via `gaia-emr init`: | Model | Size | Purpose | | ----------------------------- | -------- | ----------------------------------------- | | Qwen3.5-35B-A3B-GGUF | 18.6 GB | LLM for chat queries and patient search | | Gemma-4-E4B-it-GGUF | \~3 GB | Vision language model for form extraction | | user.embeddinggemma-300m-GGUF | \~334 MB | Embedding model for similarity search | **Disk Space:** Ensure you have at least 25 GB of free disk space for model downloads. ## Prerequisites Complete the [Setup](/docs/setup) guide first to install Lemonade Server and `uv`. Then install GAIA with the EMR extras in a Python 3.12 virtual environment. ```powershell theme={null} uv venv .venv --python 3.12 .\.venv\Scripts\Activate.ps1 uv pip install "amd-gaia[api,rag]" gaia-emr --help ``` If activation fails with a script execution error, run once: `Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser`, then retry. ```bash theme={null} uv venv .venv --python 3.12 source .venv/bin/activate uv pip install "amd-gaia[api,rag]" gaia-emr --help ``` For contributors. Clone the repo and install in editable mode (changes take effect without reinstalling): ```bash theme={null} git clone https://github.com/amd/gaia.git cd gaia uv venv .venv --python 3.12 source .venv/bin/activate # Windows: .\.venv\Scripts\Activate.ps1 uv pip install -e ".[dev,api,rag]" gaia-emr --help ``` The `api` extra provides FastAPI and uvicorn for the web dashboard; the `rag` extra provides PyMuPDF for PDF processing. `uv` downloads Python 3.12 automatically if it isn't already installed. **Having issues?** Check the [Troubleshooting](/docs/reference/troubleshooting) guide, [create an issue](https://github.com/amd/gaia/issues) on GitHub, or contact us at [gaia@amd.com](mailto:gaia@amd.com). *** ## Quick Start ### Step 1: Initialize (First Time Only) Download and load all required models before first use: ```bash theme={null} gaia-emr init ``` This command: * Checks Lemonade server is running and context size is configured * Downloads and loads all required models: * **VLM**: Gemma-4-E4B-it-GGUF (form extraction) * **LLM**: Qwen3.5-35B-A3B-GGUF (chat/query processing) * **Embedding**: user.embeddinggemma-300m-GGUF (similarity search) * Verifies all models are loaded and ready **Context Size:** For best results, set Lemonade context size to 32768. Right-click the Lemonade tray icon → Settings → Context Size → 32768. **Partial Success:** If the LLM fails to download but VLM succeeds, form extraction will still work. Chat queries and natural language patient search require the LLM. Run `gaia-emr init` again to retry failed downloads. ### Step 2: Launch ### Download sample forms [Download sample intake forms from GitHub](https://github.com/amd/gaia/tree/main/hub/agents/python/emr) and save them to a local directory (e.g., `./intake-forms/`). **Dev Install:** See the EMR agent source at `hub/agents/python/emr/` for sample forms and configuration. ### Start watching for forms ```bash theme={null} gaia-emr watch --watch-dir ./intake-forms ``` The agent will process the sample forms and display extracted patient data. ``` +-----------------------------------+ | Medical Intake Agent | | Automatic Patient Form Processing | +-----------------------------------+ Watch folder: ./intake-forms Database: ./data/patients.db File Size Hash Status -------------- -------- ------------ ------ IMG_2992.jpg 1.7 MB eaabe23e... new IMG_2993.jpg 887.6 KB 03f2391e... new IMG_2995.jpg 2.3 MB 348071a9... new IMG_2996.jpg 2.1 MB 7a73ea84... new Processing Pipeline (7 steps) File: IMG_2992.jpg [5/7] Extracting patient data VLM extracting from image... Extracted 1844 chars in 14.12s Pipeline complete in 14.3s -> Alice Williams Extracted Fields Identity first_name Alice last_name Williams date_of_birth 1980-04-04 Contact phone (411) 413-1234 email alice.williams@hotmail.com Insurance insurance_provider Medicaid Demo ... 34 fields extracted ``` ### Query patients Once processing completes, you can query the database using natural language. The agent uses tool calling to translate your questions into SQL queries and retrieve results from the SQLite database. ``` Which patients have allergies? ``` ``` Show me all patients processed today ``` ``` Summarize today's intake forms ``` Type `quit` or press `Ctrl+C` to stop. ### CLI Commands Reference ```bash init theme={null} gaia-emr init ``` ```bash watch theme={null} gaia-emr watch --watch-dir ./forms ``` ```bash dashboard theme={null} gaia-emr dashboard ``` ```bash query theme={null} gaia-emr query "patients with allergies" ``` ```bash reset theme={null} gaia-emr reset ``` ```bash help theme={null} gaia-emr -h ``` | Command | Description | | ----------- | ------------------------------------------------------------ | | `init` | Download all required models (VLM, LLM, embedding) | | `watch` | Watch folder and process forms | | `process` | Process a single form file and exit | | `dashboard` | Launch web dashboard (Electron or browser) | | `query` | One-shot database query | | `stats` | Print database statistics (patient count, file counts, etc.) | | `reset` | Delete database and start fresh | | `test` | Run end-to-end self-test against bundled samples | | `-h` | Full command reference | The recommended way to use the EMR Dashboard is via the native Electron desktop app. ### Launch the desktop app ```bash theme={null} gaia-emr dashboard ``` The dashboard opens automatically in a native desktop window. First run will install Electron dependencies automatically. The default database is `./data/patients.db`. ### Add intake forms Drag and drop intake form images or PDFs directly onto the **Watch Folder** panel in the dashboard. The agent will process them automatically. [Download sample intake forms from GitHub](https://github.com/amd/gaia/tree/main/hub/agents/python/emr) to test the agent. If you installed from source (Windows/Linux Dev tabs), you need to build the frontend first: ```bash theme={null} cd hub/agents/python/emr/gaia_agent_emr/dashboard/frontend ``` ```bash theme={null} npm install ``` ```bash theme={null} npm run build ``` Then return to the repository root: ```bash theme={null} cd ../../../../../.. ``` ### Dashboard Features The dashboard includes four main views: * **Dashboard** - Real-time stats, cumulative efficiency metrics, and live processing feed * **Patient Database** - Searchable patient list with detailed records * **Chat** - Natural language queries about patients * **Settings** - Configure watch directory, upload files, and manage database **Watch Folder Panel:** The left column displays a Watch Folder panel with status indicators: * **Green dot** - Processed files * **Red flashing dot** - Currently processing * **Orange dot** - Queued for processing **Drag-and-Drop:** Drop intake form images or PDFs directly onto the Watch Folder panel to upload and process them instantly. ### Command Options ```bash default theme={null} gaia-emr dashboard ``` ```bash --watch-dir theme={null} gaia-emr dashboard --watch-dir /path/to/forms ``` ```bash --browser theme={null} gaia-emr dashboard --browser ``` ```bash --port theme={null} gaia-emr dashboard --port 3000 ``` ```bash --no-open theme={null} gaia-emr dashboard --no-open ``` | Option | Default | Description | | ------------- | -------------------- | ------------------------------------- | | `--watch-dir` | `./intake_forms` | Directory to monitor for intake forms | | `--db` | `./data/patients.db` | SQLite database path | | `--port` | `8080` | Server port | | `--browser` | off | Open in browser instead of Electron | | `--no-open` | off | Don't auto-open, server only | If Electron/Node.js is not available, the dashboard automatically falls back to opening in your default web browser. *** ## Supported Intake Form Formats The agent accepts scanned or photographed intake forms in these formats: | Extension | Processing | | ----------------------- | ------------------------------ | | `.png`, `.jpg`, `.jpeg` | Direct image processing | | `.pdf` | Converted to image via PyMuPDF | | `.tiff`, `.bmp` | Direct image processing | *** ## Under the Hood * **Image preprocessing** — Before reaching the VLM, forms are auto-rotated from EXIF orientation (critical for phone photos), scaled to a max 1024px dimension, and JPEG-compressed (quality 85) to balance extraction quality against image token count. * **Returning-patient detection** — A multi-signal lookup combines exact name + DOB match, fuzzy matching for misspellings ("Jon Smith" → "John Smith"), and `user.embeddinggemma-300m-GGUF` vector similarity. When a returning patient is detected, changes from their previous record (new allergies, updated insurance) are highlighted. * **Real-time dashboard** — The FastAPI backend streams processing events to the React frontend over Server-Sent Events (SSE), so the UI updates within \~100ms of file detection without polling or WebSockets. * **Safe DB queries** — Natural language questions are answered via LLM tool calling: the model never writes SQL directly, it calls predefined, validated tools (e.g. `search_patients`) that construct queries safely. *** ## Troubleshooting ### Context Size Too Small ``` Context size too small! Image requires 4203 tokens but model context is only 4096. ``` Large form images can require 4,000–8,000+ tokens. Set Lemonade context size to **32768**: right-click the Lemonade tray icon → Settings → Context Size → 32768, then restart the model. ### Model Download Failed / Init Fails ``` Download succeeded but failed to rename file: The process cannot access the file ``` Run `gaia-emr init` again to resume. If it keeps failing, close any apps using the model files, delete the partial/corrupted model folder from Lemonade's cache (Windows: `%LOCALAPPDATA%\AMD\LemonadeModels\`, Linux: `~/.local/share/lemonade/models/`), restart Lemonade Server, then re-run `gaia-emr init`. ### PyMuPDF Required ``` ERROR: PyMuPDF required for PDF processing ``` Install the RAG extra: `pip install "amd-gaia[rag]"`. ### JSON Parse Failed ``` WARNING: Failed to parse extraction for: form.jpg ``` The VLM output wasn't valid JSON — usually a low-quality or unclear form image. Check image clarity, or re-run `gaia-emr test ` to inspect the raw extraction. ### Database Locked ``` ERROR: database is locked ``` Only one agent process should access a given database file at a time. Stop other `gaia-emr` processes pointed at the same `--db` path. *** ## Learn More Build this agent from scratch and understand the core components Deep dive into the web dashboard and REST API endpoints Database schema, processing pipeline, and system design *** **License** Copyright(C) 2024-2026 Advanced Micro Devices, Inc. All rights reserved. SPDX-License-Identifier: MIT