Skip to main content

Agent UI Agent Capabilities Plan

Branch: kalin/agent-ui Date: 2026-03-06 Two Milestones:
  • Milestone A — Agent UI: Wire Existing SDK Capabilities (#15) Expose existing GAIA SDK features to the Agent UI. No new SDK code — just wiring, MCP integration, and UI work.
  • Milestone B — GAIA Agent SDK: New Capabilities (TBD) Enhance the core GAIA Agent SDK with capabilities that don’t exist yet: guardrails framework, screenshot capture, computer use, voice, etc.

Milestone Scope Summary

Milestone A — Agent UI: Wire Existing SDK Capabilities

Goal: Make ChatAgent as capable as possible using what the SDK already has.

Milestone B — GAIA Agent SDK: New Capabilities

Goal: Build new capabilities in the core SDK that don’t exist anywhere today.

1. Current GAIA SDK Capability Inventory

1.1 Agents

1.2 ChatAgent Tools (Current — What the Agent UI Uses)

1.3 CodeAgent Tools (Available in SDK, NOT in Agent UI)

1.4 Other SDK Capabilities (Not Exposed to Any Agent)


2. Gap Analysis: Agent UI Agent vs. Modern PC Agent Expectations

2.1 Capabilities Users Expect Today

Based on the current landscape (Claude Computer Use, OpenAI Operator, Windows Copilot, etc.):

2.2 What Can Be Added to ChatAgent NOW (Reusing Existing SDK)

These capabilities already exist in the codebase and just need to be wired into ChatAgent:

2.3 What Requires New Development

These capabilities don’t exist anywhere in GAIA and need to be built:

3. Implementation Plan

Phase 1: Quick Wins — Wire Existing SDK into ChatAgent (1-2 weeks)

Extend ChatAgent with existing mixins from CodeAgent and other agents. Minimal new code. ChatAgent class after Phase 1:

Phase 2: Vision & Media (2-3 weeks)

Add image analysis, screenshot capture, and image generation.

Phase 3: Web & System (2-3 weeks)

Add web browsing, clipboard, and basic system tools.

Phase 4: Computer Use (4-6 weeks, separate milestone)

Full desktop automation. This is the most complex and security-sensitive phase.

Phase 5: Audio/Voice (2-3 weeks)

Wire existing Whisper ASR and Kokoro TTS into Agent UI.

4. Cross-Platform Requirements

All capabilities MUST work on Windows, Linux, and macOS:

5. Security Considerations


6. Issue Tracker

Already Created (Milestone #15)

To Create (Phase 1-5)


7. MCP Server Integration

7.1 Current MCP Infrastructure in GAIA

GAIA already has a robust MCP client infrastructure:
  • MCPClientMixin (gaia/mcp/mixin.py) — Any agent can connect to MCP servers and auto-register their tools
  • MCPClientManager — Manages multiple MCP server connections
  • Config file~/.gaia/mcp_servers.json for persistent server configuration
  • MCPAgent base class — agents/base/mcp_agent.py
  • MCP Bridgegaia/mcp/mcp_bridge.py exposes GAIA as an MCP server to external tools
  • Existing integrations — Docker MCP, Blender MCP already implemented
Gap: The Agent UI has NO way to manage MCP servers. Users can’t add, remove, enable/disable, or configure MCP servers from the UI. Based on real usage data from MCP Directory and mcpservers.org:

Tier 1 — Essential (High demand, directly useful for Agent UI)

Tier 3 — Windows Desktop Automation (Key for “Computer Use”)

Tier 4 — Microsoft Ecosystem (Enterprise)

7.3 Agent UI MCP Integration Design

A) MCP Server Manager Panel (Settings)

The Agent UI Settings modal gets an “MCP Servers” tab where users can:
  1. Browse/add popular servers from a curated list (Tier 1-2 above)
  2. Add custom servers by providing command + args + env config
  3. Enable/disable servers per session or globally
  4. View connected server status (connected, tools available, errors)
  5. Configure server credentials (API keys, tokens) with secure storage

B) Backend API Endpoints

C) ChatAgent MCP Integration

When the Agent UI enables an MCP server, the backend:
  1. Calls agent.connect_mcp_server(name, config)
  2. Tools from the MCP server are auto-registered in the agent’s tool registry
  3. The agent can now use those tools in its planning/execution
  4. Tools appear in the Capabilities panel (#440)

D) Curated Server Catalog

Ship a built-in catalog (~/.gaia/mcp_catalog.json) with pre-configured popular servers:

7.4 MCP Issues to Create


8. Dependencies

9. Critical Capabilities Coverage Matrix

The following capabilities were identified as user-critical priorities. This matrix tracks exactly where each is addressed in the plan and flags gaps.

9.1 Browser Control — Needs Priority Bump

The plan buries browser automation in Phase 4g which is 8+ weeks out. But the Playwright MCP server is a Tier 1 server that works TODAY with the existing MCPClientMixin. This should be promoted to Phase 1-MCP (first sprint):
The Playwright MCP provides: navigate, click, fill forms, take screenshots, read page content — all via accessibility snapshots. This covers 90% of browser use cases.

9.2 Web Search — Needs Free Fallback

ExternalToolsMixin.search_web requires a PERPLEXITY_API_KEY environment variable. If the user doesn’t have one, web search silently fails. The plan says “Low effort” but doesn’t address this. Fix: Prioritize Brave Search MCP (@anthropic/mcp-brave-search) as the default web search. It’s free-tier capable and runs as a standard MCP server. Fall back chain should be:
  1. Brave Search MCP (free, no API key for basic usage)
  2. Perplexity (if API key available, via ExternalToolsMixin)
  3. Fetch MCP (raw URL fetch + markdown conversion as last resort)

9.3 Shell Commands — Write Capability Gap

Current ShellToolsMixin is read-only by design (whitelist: ls, cat, grep, find, etc.). This is safe but limiting — users will want:
  • mkdir — create directories
  • cp/mv — copy/move files
  • pip install — install packages
  • npm install — install node packages
Recommendation: Don’t expand the shell whitelist. Instead, rely on:
  1. FileIOToolsMixin for file write/create/edit (Phase 1a)
  2. CLIToolsMixin for broader command execution (Phase 1, with guardrails)
  3. Guardrails (#438) to confirm dangerous operations

9.4 Guardrails — The Critical Prerequisite

Issue: The plan lists guardrails (#438) as “NEXT SPRINT” but also adds FileIOToolsMixin (file write), TestingMixin (Python execution), and CLIToolsMixin (arbitrary commands) in the same sprint. This means write/execute capabilities would ship before the safety mechanism that protects against them. Mandatory fix: Guardrails (#438) MUST be implemented BEFORE or simultaneously with any write/execute capability. The implementation order should be:

9.5 Email & Calendar — New Capability Needed

Completely missing from the plan. This is a critical gap for a PC agent. Users expect to triage emails, manage calendar events, and get summaries of their day.

MCP Servers Available

User Workflows (Email)

Priority

Add to Phase 1-MCP (Tier 2 servers) — these are MCP integrations, not native code. The ChatAgent just needs MCPClientMixin and the user adds the server in Settings. The plan covers low-level computer use (Phase 4: pyautogui, pywinauto) but misses the high-level application control pattern that users actually want. Users don’t say “move mouse to (432, 128) and click” — they say “play my Discover Weekly on Spotify” or “open my latest document in Word”.

MCP Servers for Application Control

CUA (Computer Use Agent) Strategy

Two complementary approaches:
  1. MCP-first (preferred): Use app-specific MCP servers for structured, reliable control. Spotify MCP is better than clicking the Spotify UI because it’s API-driven, reliable, and doesn’t break when the UI changes.
  2. Vision + automation (fallback): For apps without MCP servers, use:
    • Screenshot → VLM (identify UI elements) → pyautogui (click/type)
    • This is Phase 4 in the plan and requires VLM + guardrails

Demo Workflows


10. Detailed Plan Critique & Issues Found

10.1 CRITICAL: FileIOToolsMixin Has Hidden Dependency

Plan says: “Add FileIOToolsMixin to ChatAgent — Low effort, just add mixin” Reality: FileIOToolsMixin has a hard dependency on ValidationAndParsingMixin. From src/gaia/agents/code/tools/file_io.py lines 26-31:
When read_file processes a .py file (line 99), it calls self._validate_python_syntax(content) which is defined in ValidationAndParsingMixin, not in FileIOToolsMixin. Without it, reading ANY Python file will crash with AttributeError. Impact: Effort is Medium, not Low. Options:
  1. Add ValidationAndParsingMixin to ChatAgent (drags in CodeSymbol, ParsedCode models, validator classes)
  2. Refactor FileIOToolsMixin to make Python validation optional (try/except around _validate_python_syntax)
  3. Create a lightweight ChatFileIOToolsMixin that strips out Python-specific features
Recommendation: Option 2 — refactor with graceful degradation:

10.2 CRITICAL: _TOOL_REGISTRY Is Global — Tool Count Explosion

The plan proposes adding 6+ mixins to ChatAgent. Each mixin registers tools into _TOOL_REGISTRY which is a module-level global dict (src/gaia/agents/base/tools.py:16). Current tool counts:
  • ChatAgent: ~13 tools (12 @tool decorators across 3 files)
  • CodeAgent: ~57 tools (69 @tool decorators across 12 files, minus register functions)
If we add FileIOToolsMixin (11 tools), CLIToolsMixin (6 tools), ExternalToolsMixin (3 tools), ProjectManagementMixin (4 tools), TestingMixin (3 tools), plus MCP tools (variable), ChatAgent could have 40+ tools. Problem: Every tool’s full docstring gets appended to the system prompt via _format_tools_for_prompt() (agent.py:370-384). With 40 tools averaging 10 lines of description each, that’s 400+ lines of tool descriptions in the system prompt. The default context window is min_context_size: 32768 tokens (~24K words). Tool descriptions alone could consume 15-25% of the context. Impact:
  • Reduced context for actual conversation history
  • LLM confusion from too many tool choices (decision paralysis)
  • Slower inference (more tokens to process)
Recommendations:
  1. Lazy tool loading: Only register tools when their category is needed (e.g., don’t load Prisma tools if user isn’t doing database work)
  2. Tool description compression: Use 1-2 sentence descriptions in prompts, not full docstrings
  3. Tool categories: Group tools and let the LLM request a category expansion
  4. Per-session tool selection: Let users enable/disable tool categories from the UI (ties into #440 Agent Capabilities Discovery)

10.3 HIGH: ExternalToolsMixin — Silent Failure Risk

ExternalToolsMixin imports from gaia.mcp.external_services:
  • get_context7_service() — requires npx on PATH (Node.js installed)
  • get_perplexity_service() — requires PERPLEXITY_API_KEY env var
If neither is available, both tools will return error results but the tools are still registered in the system prompt. The LLM will repeatedly try to use them and fail. Fix: Conditional tool registration — only register tools if their backend is available:

10.4 MEDIUM: MCP-vs-Native Build/Buy Confusion

The plan proposes BOTH native implementations AND MCP server equivalents for the same capabilities: The plan doesn’t resolve which to use. Having both creates:
  • Duplicate tools in registry (LLM sees read_file AND filesystem__read_file)
  • Conflicting behaviors (different error formats, different security models)
  • Maintenance burden
Recommendation: Clear decision framework:
  • Native tools for core, always-available capabilities (file I/O, shell)
  • MCP servers for external integrations and optional capabilities (GitHub, Spotify, email)
  • MCP preferred when the MCP server is more capable (Playwright MCP > building our own)
  • Never both for the same capability in the same session

10.5 MEDIUM: Missing Effort Estimates & Timeline Reality

The plan says “Phase 1: 1-2 weeks” but doesn’t account for:
  • Guardrails framework (#438) — design + implement + test = 1 week minimum alone
  • FileIOToolsMixin refactoring (§10.1) — 2-3 days
  • MCP Server Manager UI — new Settings tab + API endpoints + state management = 1 week
  • Testing all new tools on Windows + Linux = 3-5 days
Realistic timeline:

10.6 LOW: Cross-Platform Testing Gaps

The plan’s cross-platform table (§4) lists tools but doesn’t mention:
  • CI/CD: No GitHub Actions matrix for Windows/Linux/macOS testing
  • pyautogui on headless: Won’t work in CI without virtual display
  • macOS permissions: Screenshot, accessibility, and automation all require explicit System Preferences permissions that can’t be automated

11. User Workflow Validation Tests

Every major capability should have a user workflow — a concrete end-to-end scenario that validates the capability works. These serve as acceptance criteria and demo scripts.

11.1 File Operations Workflows

11.2 Web & Search Workflows

11.3 Document Analysis Workflows

11.4 Shell & System Workflows

11.5 Email & Calendar Workflows

11.6 Browser & App Control Workflows

11.7 Guardrails Validation Workflows


12. Updated MCP Server Catalog (Complete)

Adding the missing Tier 2+ servers identified in the critique:

Tier 2+ — Communication & Productivity (NEW)


13. New SDK Capabilities: MCP Auto-Discovery & SKILL.md Support

13.1 MCP Server Auto-Discovery & Installation

Problem: When a user asks the agent to do something it can’t (e.g., “check my email”), the agent currently says “I can’t do that.” A modern agent should be able to find, recommend, and install the right MCP server to gain the capability. Design:
Implementation: Curated Capability Map (ships with GAIA):

13.2 Anthropic SKILL.md Support for GAIA Agent SDK

What is SKILL.md? Anthropic’s specification for agents to document their learned skills — reusable procedures, workflows, and domain knowledge that persist across sessions. Skills are stored as markdown files that the agent can read, update, and reference. Why it matters: GAIA agents should be able to learn from experience. If a user teaches the agent a multi-step workflow (“here’s how to deploy our app”), that knowledge should persist and be reusable. Design for GAIA: SKILL.md Format (Anthropic-compatible):
Integration with existing GAIA features:
  • Skills can reference RAG documents (“See indexed doc: architecture.pdf”)
  • Skills can reference MCP servers (“Requires: gmail-mcp-server”)
  • Skills can include tool sequences that the agent replays
  • Skills directory is auto-indexed by RAG for semantic search

Milestone A — Agent UI: Wire Existing SDK (Weeks 1-6)

Milestone B — GAIA Agent SDK: New Capabilities (Weeks 3-12+)

Milestone Dependency Map

Key dependency: FileIOToolsMixin (Milestone A, week 3) needs guardrails (Milestone B, week 3) to be safe for write operations. These should be developed in parallel with guardrails landing first or simultaneously.