Agent UI Agent Capabilities Plan
Branch:kalin/agent-uiDate: 2026-03-06 Two Milestones:
- Milestone A — Agent UI: Wire Existing SDK Capabilities (#15) Expose existing GAIA SDK features to the Agent UI. No new SDK code — just wiring, MCP integration, and UI work.
- Milestone B — GAIA Agent SDK: New Capabilities (TBD) Enhance the core GAIA Agent SDK with capabilities that don’t exist yet: guardrails framework, screenshot capture, computer use, voice, etc.
Milestone Scope Summary
Milestone A — Agent UI: Wire Existing SDK Capabilities
Goal: Make ChatAgent as capable as possible using what the SDK already has.Milestone B — GAIA Agent SDK: New Capabilities
Goal: Build new capabilities in the core SDK that don’t exist anywhere today.1. Current GAIA SDK Capability Inventory
1.1 Agents
1.2 ChatAgent Tools (Current — What the Agent UI Uses)
1.3 CodeAgent Tools (Available in SDK, NOT in Agent UI)
1.4 Other SDK Capabilities (Not Exposed to Any Agent)
2. Gap Analysis: Agent UI Agent vs. Modern PC Agent Expectations
2.1 Capabilities Users Expect Today
Based on the current landscape (Claude Computer Use, OpenAI Operator, Windows Copilot, etc.):2.2 What Can Be Added to ChatAgent NOW (Reusing Existing SDK)
These capabilities already exist in the codebase and just need to be wired into ChatAgent:2.3 What Requires New Development
These capabilities don’t exist anywhere in GAIA and need to be built:3. Implementation Plan
Phase 1: Quick Wins — Wire Existing SDK into ChatAgent (1-2 weeks)
ExtendChatAgent with existing mixins from CodeAgent and other agents. Minimal new code.
ChatAgent class after Phase 1:
Phase 2: Vision & Media (2-3 weeks)
Add image analysis, screenshot capture, and image generation.Phase 3: Web & System (2-3 weeks)
Add web browsing, clipboard, and basic system tools.Phase 4: Computer Use (4-6 weeks, separate milestone)
Full desktop automation. This is the most complex and security-sensitive phase.Phase 5: Audio/Voice (2-3 weeks)
Wire existing Whisper ASR and Kokoro TTS into Agent UI.4. Cross-Platform Requirements
All capabilities MUST work on Windows, Linux, and macOS:5. Security Considerations
6. Issue Tracker
Already Created (Milestone #15)
To Create (Phase 1-5)
7. MCP Server Integration
7.1 Current MCP Infrastructure in GAIA
GAIA already has a robust MCP client infrastructure:MCPClientMixin(gaia/mcp/mixin.py) — Any agent can connect to MCP servers and auto-register their toolsMCPClientManager— Manages multiple MCP server connections- Config file —
~/.gaia/mcp_servers.jsonfor persistent server configuration MCPAgentbase class —agents/base/mcp_agent.py- MCP Bridge —
gaia/mcp/mcp_bridge.pyexposes GAIA as an MCP server to external tools - Existing integrations — Docker MCP, Blender MCP already implemented
7.2 Most Popular MCP Servers (2026 Ecosystem)
Based on real usage data from MCP Directory and mcpservers.org:Tier 1 — Essential (High demand, directly useful for Agent UI)
Tier 2 — High Value (Popular integrations users commonly request)
Tier 3 — Windows Desktop Automation (Key for “Computer Use”)
Tier 4 — Microsoft Ecosystem (Enterprise)
7.3 Agent UI MCP Integration Design
A) MCP Server Manager Panel (Settings)
The Agent UI Settings modal gets an “MCP Servers” tab where users can:- Browse/add popular servers from a curated list (Tier 1-2 above)
- Add custom servers by providing command + args + env config
- Enable/disable servers per session or globally
- View connected server status (connected, tools available, errors)
- Configure server credentials (API keys, tokens) with secure storage
B) Backend API Endpoints
C) ChatAgent MCP Integration
- Calls
agent.connect_mcp_server(name, config) - Tools from the MCP server are auto-registered in the agent’s tool registry
- The agent can now use those tools in its planning/execution
- Tools appear in the Capabilities panel (#440)
D) Curated Server Catalog
Ship a built-in catalog (~/.gaia/mcp_catalog.json) with pre-configured popular servers:
7.4 MCP Issues to Create
8. Dependencies
9. Critical Capabilities Coverage Matrix
The following capabilities were identified as user-critical priorities. This matrix tracks exactly where each is addressed in the plan and flags gaps.9.1 Browser Control — Needs Priority Bump
The plan buries browser automation in Phase 4g which is 8+ weeks out. But the Playwright MCP server is a Tier 1 server that works TODAY with the existingMCPClientMixin. This should be promoted to Phase 1-MCP (first sprint):
9.2 Web Search — Needs Free Fallback
ExternalToolsMixin.search_web requires a PERPLEXITY_API_KEY environment variable.
If the user doesn’t have one, web search silently fails. The plan says “Low effort”
but doesn’t address this.
Fix: Prioritize Brave Search MCP (@anthropic/mcp-brave-search) as the
default web search. It’s free-tier capable and runs as a standard MCP server. Fall
back chain should be:
- Brave Search MCP (free, no API key for basic usage)
- Perplexity (if API key available, via ExternalToolsMixin)
- Fetch MCP (raw URL fetch + markdown conversion as last resort)
9.3 Shell Commands — Write Capability Gap
CurrentShellToolsMixin is read-only by design (whitelist: ls, cat, grep,
find, etc.). This is safe but limiting — users will want:
mkdir— create directoriescp/mv— copy/move filespip install— install packagesnpm install— install node packages
FileIOToolsMixinfor file write/create/edit (Phase 1a)CLIToolsMixinfor broader command execution (Phase 1, with guardrails)- Guardrails (#438) to confirm dangerous operations
9.4 Guardrails — The Critical Prerequisite
Issue: The plan lists guardrails (#438) as “NEXT SPRINT” but also addsFileIOToolsMixin (file write), TestingMixin (Python execution), and CLIToolsMixin
(arbitrary commands) in the same sprint. This means write/execute capabilities would
ship before the safety mechanism that protects against them.
Mandatory fix: Guardrails (#438) MUST be implemented BEFORE or simultaneously
with any write/execute capability. The implementation order should be:
9.5 Email & Calendar — New Capability Needed
Completely missing from the plan. This is a critical gap for a PC agent. Users expect to triage emails, manage calendar events, and get summaries of their day.MCP Servers Available
User Workflows (Email)
Priority
Add to Phase 1-MCP (Tier 2 servers) — these are MCP integrations, not native code. The ChatAgent just needsMCPClientMixin and the user adds the server in
Settings.
9.6 Application Control (CUA) & Popular App Demos
The plan covers low-level computer use (Phase 4: pyautogui, pywinauto) but misses the high-level application control pattern that users actually want. Users don’t say “move mouse to (432, 128) and click” — they say “play my Discover Weekly on Spotify” or “open my latest document in Word”.MCP Servers for Application Control
CUA (Computer Use Agent) Strategy
Two complementary approaches:- MCP-first (preferred): Use app-specific MCP servers for structured, reliable control. Spotify MCP is better than clicking the Spotify UI because it’s API-driven, reliable, and doesn’t break when the UI changes.
-
Vision + automation (fallback): For apps without MCP servers, use:
- Screenshot → VLM (identify UI elements) → pyautogui (click/type)
- This is Phase 4 in the plan and requires VLM + guardrails
Demo Workflows
10. Detailed Plan Critique & Issues Found
10.1 CRITICAL: FileIOToolsMixin Has Hidden Dependency
Plan says: “Add FileIOToolsMixin to ChatAgent — Low effort, just add mixin”
Reality: FileIOToolsMixin has a hard dependency on ValidationAndParsingMixin.
From src/gaia/agents/code/tools/file_io.py lines 26-31:
read_file processes a .py file (line 99), it calls self._validate_python_syntax(content)
which is defined in ValidationAndParsingMixin, not in FileIOToolsMixin. Without it,
reading ANY Python file will crash with AttributeError.
Impact: Effort is Medium, not Low. Options:
- Add
ValidationAndParsingMixinto ChatAgent (drags inCodeSymbol,ParsedCodemodels, validator classes) - Refactor
FileIOToolsMixinto make Python validation optional (try/except around_validate_python_syntax) - Create a lightweight
ChatFileIOToolsMixinthat strips out Python-specific features
10.2 CRITICAL: _TOOL_REGISTRY Is Global — Tool Count Explosion
The plan proposes adding 6+ mixins to ChatAgent. Each mixin registers tools into
_TOOL_REGISTRY which is a module-level global dict (src/gaia/agents/base/tools.py:16).
Current tool counts:
- ChatAgent: ~13 tools (12 @tool decorators across 3 files)
- CodeAgent: ~57 tools (69 @tool decorators across 12 files, minus register functions)
FileIOToolsMixin (11 tools), CLIToolsMixin (6 tools), ExternalToolsMixin
(3 tools), ProjectManagementMixin (4 tools), TestingMixin (3 tools), plus MCP
tools (variable), ChatAgent could have 40+ tools.
Problem: Every tool’s full docstring gets appended to the system prompt via
_format_tools_for_prompt() (agent.py:370-384). With 40 tools averaging 10 lines of
description each, that’s 400+ lines of tool descriptions in the system prompt.
The default context window is min_context_size: 32768 tokens (~24K words). Tool
descriptions alone could consume 15-25% of the context.
Impact:
- Reduced context for actual conversation history
- LLM confusion from too many tool choices (decision paralysis)
- Slower inference (more tokens to process)
- Lazy tool loading: Only register tools when their category is needed (e.g., don’t load Prisma tools if user isn’t doing database work)
- Tool description compression: Use 1-2 sentence descriptions in prompts, not full docstrings
- Tool categories: Group tools and let the LLM request a category expansion
- Per-session tool selection: Let users enable/disable tool categories from the UI (ties into #440 Agent Capabilities Discovery)
10.3 HIGH: ExternalToolsMixin — Silent Failure Risk
ExternalToolsMixin imports from gaia.mcp.external_services:
get_context7_service()— requiresnpxon PATH (Node.js installed)get_perplexity_service()— requiresPERPLEXITY_API_KEYenv var
10.4 MEDIUM: MCP-vs-Native Build/Buy Confusion
The plan proposes BOTH native implementations AND MCP server equivalents for the same capabilities:
The plan doesn’t resolve which to use. Having both creates:
- Duplicate tools in registry (LLM sees
read_fileANDfilesystem__read_file) - Conflicting behaviors (different error formats, different security models)
- Maintenance burden
- Native tools for core, always-available capabilities (file I/O, shell)
- MCP servers for external integrations and optional capabilities (GitHub, Spotify, email)
- MCP preferred when the MCP server is more capable (Playwright MCP > building our own)
- Never both for the same capability in the same session
10.5 MEDIUM: Missing Effort Estimates & Timeline Reality
The plan says “Phase 1: 1-2 weeks” but doesn’t account for:- Guardrails framework (#438) — design + implement + test = 1 week minimum alone
FileIOToolsMixinrefactoring (§10.1) — 2-3 days- MCP Server Manager UI — new Settings tab + API endpoints + state management = 1 week
- Testing all new tools on Windows + Linux = 3-5 days
10.6 LOW: Cross-Platform Testing Gaps
The plan’s cross-platform table (§4) lists tools but doesn’t mention:- CI/CD: No GitHub Actions matrix for Windows/Linux/macOS testing
- pyautogui on headless: Won’t work in CI without virtual display
- macOS permissions: Screenshot, accessibility, and automation all require explicit System Preferences permissions that can’t be automated
11. User Workflow Validation Tests
Every major capability should have a user workflow — a concrete end-to-end scenario that validates the capability works. These serve as acceptance criteria and demo scripts.11.1 File Operations Workflows
11.2 Web & Search Workflows
11.3 Document Analysis Workflows
11.4 Shell & System Workflows
11.5 Email & Calendar Workflows
11.6 Browser & App Control Workflows
11.7 Guardrails Validation Workflows
12. Updated MCP Server Catalog (Complete)
Adding the missing Tier 2+ servers identified in the critique:Tier 2+ — Communication & Productivity (NEW)
13. New SDK Capabilities: MCP Auto-Discovery & SKILL.md Support
13.1 MCP Server Auto-Discovery & Installation
Problem: When a user asks the agent to do something it can’t (e.g., “check my email”), the agent currently says “I can’t do that.” A modern agent should be able to find, recommend, and install the right MCP server to gain the capability. Design:
Curated Capability Map (ships with GAIA):
13.2 Anthropic SKILL.md Support for GAIA Agent SDK
What is SKILL.md? Anthropic’s specification for agents to document their learned skills — reusable procedures, workflows, and domain knowledge that persist across sessions. Skills are stored as markdown files that the agent can read, update, and reference. Why it matters: GAIA agents should be able to learn from experience. If a user teaches the agent a multi-step workflow (“here’s how to deploy our app”), that knowledge should persist and be reusable. Design for GAIA:
SKILL.md Format (Anthropic-compatible):
- Skills can reference RAG documents (“See indexed doc: architecture.pdf”)
- Skills can reference MCP servers (“Requires: gmail-mcp-server”)
- Skills can include tool sequences that the agent replays
- Skills directory is auto-indexed by RAG for semantic search
14. Summary: Recommended Priority Order (Revised, Split by Milestone)
Milestone A — Agent UI: Wire Existing SDK (Weeks 1-6)
Milestone B — GAIA Agent SDK: New Capabilities (Weeks 3-12+)
Milestone Dependency Map
FileIOToolsMixin (Milestone A, week 3) needs guardrails
(Milestone B, week 3) to be safe for write operations. These should be developed
in parallel with guardrails landing first or simultaneously.