​

Agent UI Agent Capabilities Plan

Branch: kalin/agent-ui Date: 2026-03-06 Two Milestones:

• Milestone A — Agent UI: Wire Existing SDK Capabilities (#15) Expose existing GAIA SDK features to the Agent UI. No new SDK code — just wiring, MCP integration, and UI work.
• Milestone B — GAIA Agent SDK: New Capabilities (TBD) Enhance the core GAIA Agent SDK with capabilities that don’t exist yet: guardrails framework, screenshot capture, computer use, voice, etc.

​

Milestone Scope Summary

​

Milestone A — Agent UI: Wire Existing SDK Capabilities

​

Milestone B — GAIA Agent SDK: New Capabilities

​

0. v2 Architecture: Sidecar-First (supersedes the in-process model below)

Realizes the Architecture (v2 — sidecar-first) section of agent-ui.mdx. The Milestone A/B wiring below assumes agents run in-process inside the UI backend; v2 inverts that — agents are out-of-process sidecars and the UI is a thin host. Where a section below wires a capability into in-process ChatAgent, read it as “a capability the sidecar exposes over REST and the host relays,” not “Python loaded into the UI backend.” PR #1910 (in-UI EmailProxyAgent) is superseded.

​

0.§ Reading map

• Foundations: §0.0 the host is a headless daemon (not the web UI) · §0.3 the sidecar supervisor.
• The agent contract: §0.1 REST surface (fixed-function + /query) · §0.2 /query SSE event schema · §0.4 mid-workflow confirmation · §0.18 dispatch · §0.32 multi-agent orchestration · §0.33 /query on the gaia api REST server.
• Autonomy: §0.22 the scheduler clock · §0.34 autonomy-readiness (the human-in-the-loop → policy-driven gap).
• Structure & framing: §0.35 architecture-review refinements (two-tier custody contract, naming, module seams, v1-hardening scope, render primitives) · §0.36 third-party integration ergonomics (the tiered capability matrix) · §0.37 pluggable custody (one sidecar = rich and stateless; refines §0.9) · §0.38 memory/resource footprint · §0.39 custody-modes comparison table · §0.40 hybrid custody (per-store shared/private — the embedded↔delegated spectrum) · §0.41 the unified capability-mediation model (collapses the modes — the capstone).
• Auth & security: §0.6 OAuth forward · §0.11 the three auth legs + per-agent authorization · §0.24 third-party containment (signing, tiers, egress, encrypt-at-rest, audit integrity, taint) 🔒.
• Shared resources (host custody): §0.9 memory/RAG/sessions/audit/MCP/model table · §0.12 the model-slot broker · §0.19 audit sink · §0.29 store consistency.
• Data & contracts: §0.28 the agent manifest schema · §0.29 custody-store consistency · §0.30 identifier catalog · §0.31 the /host/v1/* callback API · §0.15 contract evolution.
• Lifecycle & ops: §0.5 install (+model provisioning) · §0.13 concurrency / cancel / failure / reaper · §0.14 one daemon per machine · §0.15 version negotiation · §0.16 dev-mode · §0.20 uninstall · §0.21 offline/footprint · §0.22 autonomy clock · §0.25 daemon lifecycle · §0.26 on-disk layout.
• Delivery & context: §0.7 CLI parity · §0.8 what’s superseded · §0.10 migration (strangler-fig + data) · §0.17 eval/test · §0.23 feasibility & build order · §0.27 relationship to sibling plans.
• Open decisions needing sign-off: confirmation model (§0.4) · memory/RAG home (§0.9) · third-party trust-root + containment (§0.24) 🔒.

​

0.0 “The host” is a headless daemon, not the web UI

• CLI-only / no-UI works. gaia email "what did I say about X last week" and its audit writes reach /host/v1/* because the daemon — not the browser tab — is the custody server. Without this split, any sidecar spawned while the UI is closed would have no callback target (the exact hole §0.11/§0.19 exist to prevent).
• The daemon is the always-on component that survives the web UI closing, owns the machine-wide single-instance registry, and is the wake-up owner for autonomy (§0.22). Starting the UI or a CLI command auto-starts the daemon if not already running.

​

0.1 The agent REST contract (one per sidecar)

• Fixed-function endpoints — the deterministic core as one call each (POST /v1/<agent>/<capability>: triage, search, pre-scan, archive, calendar- create, …). No LLM in the loop; for scripted/integrator use. Not every capability is a fixed-function call — the spec’s Capability → surface table marks the advanced/personalization/detect-and-reason tier “agent only”, so those are reached via /query + the §0.22 scheduler, not a deterministic endpoint. (So “exposes ALL capabilities” is true of the sidecar, split across the two surfaces — not “every capability has a fixed endpoint.”)
• POST /v1/<agent>/query — the agent loop, exposed over REST like any other endpoint. NL request in; the sidecar reasons and chains tools into multi-step workflows; response is text/event-stream (SSE). This is the surface the UI’s chat experience and the gaia <agent> CLI both drive.
• GET /health, GET /version — readiness + contract-version handshake.
• Connector surfaces: self-service OAuth routes (bare integrators) and forwarded-connection intake POST /v1/connections gated by X-Gaia-UI: 1 (the Agent UI host path — see §0.6).

• The host mints run_id, not the sidecar. §0.13’s cancel/{run_id} and the confirmation flow key off it; if the sidecar minted it, a “Stop” pressed before the first SSE event would have no id to cancel (a race window). Host-minted run_id is cancellable from the instant the request is sent.
• Context is pushed in the body, not pulled. The host owns the transcript (§0.9) and passes the relevant slice as context; the sidecar stays stateless and never reads other sessions back over the callback (which §0.11’s scoping would forbid anyway). The GET /host/v1/sessions callback is host-internal, not a sidecar transcript-access path — so a pushed slice and a pulled one can’t disagree.

​

0.2 /query SSE event schema

​

0.3 Host: the per-agent sidecar supervisor

• GET /v1/agents — installed + catalog (hub mirror, §0.5).
• POST /v1/agents/{id}/install · DELETE /v1/agents/{id} — one-click install/uninstall.
• ANY /v1/<agent>/* — reverse-proxy to that agent’s running sidecar (lazy-start), preserving SSE for /query and the sidecar’s own status codes + actionable detail.

​

0.4 Mid-workflow confirmation (over SSE, no WebSocket)

1. Sidecar reaches a gated step, emits needs_confirmation{run_id, action, summary, confirm_url} and pauses the run (kept alive server-side, keyed by run_id).
2. UI shows approve/deny with the literal summary (recipient/subject/body, etc.).
3. On approve the host POSTs to confirm_url (/v1/<agent>/query/{run_id}/confirm) with the minted token; the sidecar resumes emitting on the same SSE stream. Deny ends the run with a final “skipped” answer.

​

0.5 Agent Hub mirror (install / uninstall)

​

0.6 OAuth: host owns consent + refresh, forwards to the sidecar

• Role inversion. Today /v1/connections is a host-side intake — external apps forward connections INTO gaia (src/gaia/ui/routers/connectors.py:94 forwarded_router; src/gaia/connectors/api.py:266 import_forwarded_connection). In v2 the host becomes the forwarder OUT, and the sidecar exposes the intake (it bundles gaia.connectors). The primitive exists; the direction and the per-agent wiring are new.
• Token refresh ownership. With N sidecars sharing one Google grant, each holding a refresh token would race to refresh and rotate each other out. The single-writer host should own refresh and forward short-lived access tokens (re-forward on expiry), so sidecars never touch the refresh token.

​

0.7 CLI ↔ sidecar parity

• memory writes would vanish when the command exits — the §0.0 hole. The daemon being the sole spawner is what closes it.)

​

0.8 What’s superseded

• In-process agent construction in src/gaia/ui/_chat_helpers.py (the agent_type=… factory) → replaced by proxying to sidecars.
• EmailProxyAgent + the in-UI tool loop (PR #1910) → replaced by relaying to the sidecar’s /query.
• Fat UI routers whose logic belongs to an agent (chat loop, scheduling) → migrate into the relevant sidecar; the host keeps UI serving, shared-user-data custody (§0.9), and sidecar supervision.

​

0.9 Shared services (host custody) — the biggest open call

Refined by §0.37: custody is a pluggable provider, not intrinsically the host. The rationale below holds for the multi-agent Agent UI (N sidecars sharing one user’s data ⇒ single writer ⇒ the host). A standalone sidecar is its own single writer and uses an embedded provider. So read this table as “the provider the GAIA host implements,” not “the only place custody can live.”

​

0.10 Migration path (strangler-fig — no big-bang)

0. Data migration (one-time, idempotent, versioned). The current ~/.gaia state predates host custody: today’s transcripts (sessions.py store) have no agent tag, and the single memory store has no host/agent-private partition. Migrate on first v2 launch: existing sessions → host session index with a default agent tag; existing memory → host user memory (the conservative default — it was cross-agent already), with a documented rule for what becomes agent-private. Stamp an on-disk schema version so the migration is detectable and runs once. Without this, upgraders lose past chats or see them reattach to a wrong/null agent, and memory either leaks cross-agent or is stranded.
1. Email first — already sidecar-shaped; add /query (SSE) and route the UI’s email session through it. All other agents remain in-process, untouched.
2. Host supervisor — generalize EmailSidecarManager → AgentSidecarManager
   • hub install/uninstall, so in-process and sidecar agents run side by side.
3. Per-agent migration — move one agent at a time behind the proxy; delete its in-process router logic only after its sidecar ships.
4. Extract shared services — lift user memory / RAG / session index into the host custody layer (§0.9) as agents stop owning them in-process.

​

0.11 Auth: three legs (client↔daemon, daemon↔sidecar, sidecar↔daemon)

• Client → daemon (new with the §0.0 split — 🔒). The daemon is now a standalone process exposing, on a loopback port, the whole custody API (/host/v1/memory|rag| sessions|audit) and the ANY /v1/<agent>/* proxy that can drive any sidecar. The UI and CLI are external clients of it, and loopback is not an auth boundary (same threat model as below) — so without a credential, any local process can read the entire cross-agent memory/RAG/transcript store or drive any sidecar through the proxy. The daemon mints a client-auth token at startup, stored 0600 in its ~/.gaia/host/instance.json (§0.14); UI and CLI read it and present it on every daemon call. (A unix-domain socket with SO_PEERCRED is a stronger alternative where available.) This is distinct from the per-spawn secret below, which is daemon↔sidecar only.
• Per-spawn secret (daemon → sidecar). Loopback binding is not an auth boundary — any local process (another user on a shared box, a browser tab via CSRF/DNS-rebinding, a malicious npm postinstall) can POST /v1/<agent>/query "archive everything" on an unauthenticated port that can send mail. The host mints a random secret per spawn, passes it to the sidecar via env at launch, and the sidecar requires it (bearer header) on every request — fixed-function, /query, and /confirm. X-Gaia-UI: 1 is CSRF hygiene, not authentication. (In the daemon-less bare-integrator topology of §0.6, the process that launches the sidecar supplies this launch secret — the sidecar always requires it; only the injector changes.)
• The reverse contract (sidecar → host). The custody model (§0.9) needs sidecars to read host-owned data. Define a first-class host callback API — GET /host/v1/memory, POST /host/v1/rag/query, GET /host/v1/sessions/{id} — that the sidecar calls, authenticated with the same per-spawn secret (the host injects its own callback base-URL + the secret at launch). Symmetric to the forward proxy; without it, “the host stores it, the sidecar queries it” has no transport and agents silently re-fork their own copies.
• Authorization, not just authentication (🔒 the decisive one). The per-spawn secret only proves “the host spawned me” — it does not say which agent, and unscoped callback routes turn the custody store into a single-reader exfiltration surface: a hub-installed third-party agent could read the entire cross-agent user memory, the whole RAG corpus, and any session transcript by id — including other agents’ conversations. The host must bind the secret to the agent id at mint time and scope every callback per-agent: /host/v1/memory and /host/v1/rag return only rows tagged to that agent (or an explicitly user-granted shared scope), and /host/v1/sessions/{id} verifies the session belongs to the caller. Shared-memory/RAG read access becomes a per-agent grant surfaced at install, like connector grants. This is an architecture-level security decision — settle it before the callback contract is frozen.
• Secret delivery. Passing the secret via env is readable by other same-user processes on some OSes — the exact “another local process” threat the auth exists to stop. Prefer a pipe/inherited fd or a short-lived 0600 file over the environment.

​

0.12 Shared model backend — a host-owned model-slot broker

• All model families, not just the chat LLM. Host-custody RAG needs the embedder to serve /host/v1/rag/query; a VLM agent needs the VLM; voice (host I/O) needs Whisper/Kokoro; SD needs SDXL. A host RAG call mid-/query evicting the chat model is the exact silent-ctx-cap regression #1030 warns about. The lease must cover embedder + LLM + VLM (+ voice/SD) together for a run — host-custody RAG/VLM/voice are broker clients, not privileged bypass paths.
• Priority + preemption. Serialization prevents corruption, not stalls. A Phase-C autonomous brief runs in-sidecar and also draws the slot, so it can block the user’s interactive turn behind it. The broker needs interactive > background priority (and preemption), or a background job silently makes a chat turn wait.
• Affinity + legibility. A hot-model affinity hint keeps agents sharing a model from reloading on every switch; the host emits a switching model… status event so an unavoidable evict+reload stall (seconds, worse on NPU) is a visible state, not a frozen UI.

​

0.13 Concurrency, cancellation & failure semantics

• Run isolation. §0.3 mandates one shared sidecar per agent, but a /query run is a stateful multi-step loop and today’s confirmation store is a single process-wide dict. Two simultaneous callers (UI + a scheduled brief, two tabs) must not collide: namespace all per-run state by run_id (token store, tool state), forbid shared mutable module state, and document a parallelism limit (or a per-run worker). “Shared instance” must not silently imply “safely reentrant.”
• Cancellation. Define POST /v1/<agent>/query/{run_id}/cancel; the host propagates a UI “Stop” and a dropped SSE connection to it (a relayed stream hides the client disconnect from the sidecar), and the agent loop checks a cancel flag between tool steps — so a stopped run stops burning the model slot and fires no further tools.
• Failure semantics. On sidecar crash mid-SSE the host injects a synthetic terminal error event (the UI otherwise sees a truncated stream with no final), cleans up the orphaned run_id/session-index entry, and surfaces the actionable cause. Download/spawn failure fails loud via the hub (§0.5).
• Lifecycle/resource limits. Every installed agent is a long-lived process holding a port + memory + a model-slot claim. Lazy-spawn is not enough: add an idle-timeout / LRU reaper and a cap on concurrent live sidecars, or ten installed agents = ten resident processes.

​

0.14 One daemon per machine; the daemon holds one sidecar per agent

• One daemon per machine. A daemon lockfile/registry (~/.gaia/host/instance.json → pid + client-auth token — see §0.11) makes the first UI/CLI invocation start the daemon and every later one attach. Two daemons would re-create two writers into grants.json and two forwarders, so the lock is what guarantees a single forwarder-of-record.
• One sidecar per agent, owned by the daemon. The daemon keeps its own per-agent registry (AgentSidecarManager, §0.3) → pid + ephemeral port + per-spawn secret; UI and CLI never see it, they go through the daemon’s proxy. So “one shared instance per agent” is now a property of the single daemon, not a race between client processes.

​

0.15 Contract-version negotiation at install + render fallback

• Reverse callback skew (new daemon → old sidecar). §0.25 versions the client↔daemon API; the mirror case — an old installed sidecar calling a changed /host/v1/* — is uncovered. Version the callback API too, symmetric to §0.25: refuse + prompt reinstall on a MAJOR the daemon can’t serve, or hold N-1 compat.
• Unknown SSE event type (§0.2). A newer agent emitting a new top-level event type to an older host must surface a visible “unsupported event,” never silently drop it (the CLAUDE.md no-silent-fallback rule) — parity with the render-type fallback above.
• Additive vs deprecation. A new endpoint/capability/event is a backward-compatible MINOR; removing one needs a stated deprecation/sunset window, not a silent break. (The MAJOR gate alone can’t express either.)

​

0.16 Dev-mode discovery for unpublished agents

​

0.17 Testing & eval strategy (/query + the distributed seams)

• /query behavior (eval). A sidecar eval harness drives /query over REST and asserts on the event sequence (tool_call → tool_result → needs_confirmation → final), not just a final string; eval baselines move into each agent’s hub package. Runs stay serial against the shared broker (§0.12) — the per-agent sidecar model must not reintroduce concurrent model-slot contention.
• The distributed seams (deterministic unit/integration, no LLM):
  • SSE relay — streaming passthrough (assert no buffering), cancel-on-disconnect (§0.13), and the synthetic-crash error event, against a fake sidecar.
  • Auth legs (§0.11) — assert all three reject a missing/wrong credential (client→daemon token, daemon→sidecar per-spawn secret, sidecar→daemon callback), and that callback authorization is per-agent scoped (agent A cannot read agent B’s memory/session) — the security property, tested as a boundary, not a mock that only proves “we called it.”
  • Broker (§0.12) — a concurrency test proving two agents requesting different models serialize rather than race-evict (the property CLAUDE.md’s serial-eval rule protects), plus priority (interactive preempts background).
  • Daemon lifecycle (§0.25) — stale-instance.json reclaim after a killed pid, atomic write, and daemon restart-on-version-skew draining in-flight runs.
• Data migration (§0.10 step 0) — idempotency + cold-state test (CLAUDE.md’s “test from the user’s real initial state”): run the one-time migration twice, and from a real pre-v2 ~/.gaia fixture, asserting transcripts/memory land tagged correctly and a second run is a no-op.
• End-to-end golden path — one on-hardware test per the gaia-testing tiers: UI (or CLI) → daemon → sidecar → Lemonade, a real /query inbox triage, confirming the email_pre_scan card renders and a destructive step surfaces the confirmation gate. This is the “the call is valid, not just invoked” proof for the whole chain.

​

0.18 Dispatch — which sidecar answers a free-form message

• (a) Explicit per-session agent selection (recommended v1) — the UI’s active- agent picker names the sidecar; matches the CLI; zero extra LLM cost.
• (b) A routing sidecar the host calls first — flexible, but routing is itself an LLM loop, so it draws the model slot (§0.12) and adds a hop before every turn.

​

0.19 Audit trail — a host-custody sink, not agent-private

​

0.20 Uninstall — data lifecycle

• Forwarded connection — the host withdraws/revokes the OAuth connection it forwarded, so a removed agent doesn’t retain live mailbox access.
• Host-custody transcripts tagged to the agent — offer keep or delete (they survive the binary; dangling “ghost” sessions that can’t reopen are the failure to avoid); on reinstall, keep = reattach.
• Agent-private state (~/.gaia/agents/<id>/, e.g. the action log) — wipe by default (state a default), with an option to retain.
• Shared user-memory rows the agent wrote — governed by its §0.11 grant scope.

​

0.21 Offline / sideload install + install footprint

• Sideload. Every §0.5 path downloads from the lock’s baseUrl — but GAIA is privacy-first and targets air-gapped + OEM-preloaded deployments (a stated Phase-D goal). Add a local install source: point the manager at a pre-staged directory of verified binaries (still SHA-check against the lock, skip the network), covering the first agent too. This also unblocks OEM factory-image bundling — cheap to reconcile now rather than retrofit.
• Footprint (honest downgrade). The email binary is ~90 MB only because its freeze --exclude-modules torch/transformers/faiss/scipy/pandas — legal solely because email runs no in-process ML (it calls Lemonade over HTTP; packaging/freeze.py). Any agent that needs an in-process embedder/VLM/SD (RAG, vision) cannot use those excludes → multi-GB per binary. And PyInstaller produces self-contained bundles with no cross-binary linking, so a “shared runtime layer the per-agent binaries link against” is not a PyInstaller capability — it needs a different packaging strategy (a shared base venv, a host-side model-runtime service the sidecars call, or non-frozen deployment). Treat §0.21 shared-runtime as aspirational, not a config tweak, and prefer keeping in-process ML out of sidecars (route it through the host model broker, §0.12) to preserve the lightweight-binary property.

​

0.22 Autonomy — the host holds the clock, the sidecar does the work

• The host daemon (§0.0) owns the trigger registry + cron clock — always-on, it is the wake-up owner. Schedule metadata is host custody.
• At fire time the host spawns the owning sidecar (if not resident) and hands it the job over /query (or a fixed endpoint), then lets the reaper reclaim it after.
• Alternatively, mark specific agents pinned-resident (exempt from the reaper + live-cap) when sub-minute latency matters. State which agents qualify; default is wake-on-fire, since holding every autonomous agent resident defeats §0.13.

​

0.23 Feasibility & build sequencing (grounded in the current code)

• The agent-loop → SSE seam (src/gaia/ui/sse_handler.py SSEOutputHandler) — /query reuses it (§0.2 build note), not net-new instrumentation.
• Mid-workflow confirmation over SSE — the pause/resume primitive §0.4 describes (_confirm_event, _user_input_queue, permission_request events) ships in-process today; the only new work is carrying the Event across the process boundary (which is exactly what makes a resumed run stateful — the §0.4 tension is real, not hypothetical).
• OAuth forward + short-lived-token refresh — import_forwarded_connection and get_access_token (src/gaia/connectors/api.py) already exist and the refresh engine is client-neutral, so §0.6’s “host refreshes, forwards short-lived tokens” maps onto real code.

1. Lemonade broker (§0.12) — the critical path. Today the only lock is an in-process _downloads_lock (lemonade_client.py); “one model slot” literally means last-writer-wins across processes. The host broker (serialized loads + cross-process lease + priority) has zero existing scaffolding and gates the plan’s headline (more than one concurrently-active agent). Build + prove first.
2. Streaming reverse-proxy (§0.3, §0.13) — net-new httpx streaming + cancel; the current proxy buffers. Prove with a one-endpoint freeze spike (SSE through a PyInstaller binary is unproven here, though low-risk — uvicorn loops/protocols are collected in the freeze) before committing.
3. /query endpoint + event-vocab translation (§0.1, §0.2) — de-risked by the SSE seam, but the sidecar has no /query today and the handler’s vocabulary must be translated to the §0.2 contract.
4. CLI rewrite (§0.7) — gaia email is fully in-process (cli.py builds EmailTriageAgent and calls process_query); converting it to a daemon client is a real rewrite that must stay behaviorally identical.
5. Forward-OUT intake on the sidecar (§0.6) — primitive exists; the sidecar /v1/connections intake route + role inversion is new wiring (low risk).

​

0.24 Security & privacy — containing a sidecar that is itself the adversary

• Sign the lock/catalog — SHA-256 alone is not authentication (🔒). The hash proves a binary matches the lock; it says nothing about who wrote the lock. A controlled catalog/baseUrl/CDN can serve attacker code + a matching hash, or a known-vuln older version. Sign the lock/catalog with an AMD key and verify the signature before trusting any hash inside it; add anti-rollback (record highest-installed version, refuse silent downgrade); third-party publishers get a publisher signature + TOFU pin.
• Tier-gate + least-privilege the OAuth forward (🔒). §0.5 one-click + §0.6 forward means a third-party agent can be handed a live mailbox token. Nothing consults the Phase-D trust tiers (Verified/Community/Experimental) before forwarding, and the forward hands the whole grant. Make forwarding + shared scopes tier-aware, require an explicit install consent naming the exact OAuth scopes forwarded, and forward the minimum scope the agent declares.
• Constrain sidecar network egress (🔒 the decisive containment gap). A hostile sidecar holds a mailbox token and, as an ordinary process, unrestricted outbound network — it can read mail through the sanctioned connection and POST it anywhere, invisible to the callback-authorization model (which guards host custody, not the agent’s own sockets). This is the difference between “sandboxed agent” and “trusted arbitrary code with your mailbox,” and it negates “100% local.” No-network-by-default + a declared, install-surfaced egress allowlist (the manifest names hosts, e.g. googleapis.com) enforced via a host-controlled network namespace/proxy. Carve-out (do not sever the control channel): the daemon’s own loopback endpoints — the §0.11 callback API, the §0.12 broker, and the reverse-proxy — are an always-allowed control channel, separate from the external-host egress allowlist. The proxy enforcement variant satisfies this automatically (it is the host); a network-namespace variant must explicitly plumb the daemon socket into the namespace, or it would cut the very callback/broker paths the custody model depends on.
• Encrypt data at rest — but separate the two secret classes (don’t break §0.11). 0600 stops other users, not the threats that matter on a single-user desktop (stolen laptop, synced backup, same-user malware). Two distinct classes:
  • Durable custody secrets — OAuth refresh tokens (grants.json) and the custody stores (memory/RAG/transcripts) → OS keychain (Keychain/DPAPI/ libsecret) + encrypt at rest. Pull Phase-C’s “encrypted credential vault” forward to whenever §0.6 lands, not v0.23. These are the high-value, long-lived secrets a stolen disk exposes.
  • The §0.11 client-auth token (instance.json) — must stay client-readable (UI/CLI read it to call the daemon), so it legitimately remains 0600 in instance.json: it is ephemeral, re-minted every daemon startup, so losing it is harmless. Do not move it to the keychain — that would break the read path §0.11 relies on. (The daemon↔sidecar per-spawn secret is likewise ephemeral; keychain is for durable secrets only.)
• Make the audit log tamper-evident. §0.19 appends actions to a host sink, but nothing enforces append-only — a compromised process can rewrite/truncate it, defeating the observability promise. Hash-chain / rolling-MAC each entry (sealing the prior), restrictive perms, write-only exposure to agents, gap/rewrite detection surfaced in the dashboard.
• Cross-agent prompt-injection taint (name it in §0.18 now). The email agent hardens its own body-as-data, but the cross-agent path (orchestrator, “summarize this and email it”) carries untrusted mailbox content from agent A into agent B’s /query, where it can drive B’s destructive action — compounded by §0.4’s re-plan divergence. Taint must travel with the data across the boundary (mark cross-agent-sourced context untrusted so the receiver treats it as data), with the confirmation gate as backstop. A hard requirement for the orchestrator, not v1.
• MCP auto-install must not reintroduce the supply-chain hole. §0.11 names “a malicious npm postinstall” as a threat, yet §13.1 proposes auto-installing mcp-server-* from npm/GitHub on demand — unpinned/unsigned npm runs arbitrary postinstall code. Restrict auto-discovery to a curated, version-pinned, integrity-checked allowlist; never auto-execute an unpinned package; disable lifecycle scripts; gate installs behind the same signature/tier model as agents.
• Signed updates + re-consent. §0.5 specs install/uninstall but not update. Silent catalog auto-update is the classic vector (push malicious “update” to an agent already granted mailbox + memory scope). Make update a first-class, signature-verified op (reusing the signed lock), with anti-rollback and re-consent when the new version widens declared scopes/egress.
• Telemetry vs “100% local.” Any phone-home (incl. Phase-D “cost savings telemetry”) punctures the privacy promise and rides the same egress as above. Local-only aggregation by default; any network telemetry is explicit opt-in with a visible disclosure. State the default here.

​

0.25 The daemon’s OWN lifecycle — birth, death, control, update

• Birth + rebirth (fixes the §0.0↔§0.22 contradiction). §0.0 says the daemon “auto-starts on the first UI/CLI call,” but §0.22 makes it the always-on cron clock — after an overnight reboot with no human present, nothing starts it and the 8am brief never fires. Register the daemon with the OS process manager (launchd/LaunchAgent, Windows Scheduled Task/service, systemd user unit) so it starts at login/boot and restarts on crash — a supervisor-of-the-supervisor. The Phase-C system-tray app is the natural long-term home, but it lands two phases after the daemon ships, so name the interim manager. On restart, define whether pinned-resident sidecars (§0.22) are respawned or left to lazy-spawn.
• Stale instance.json recovery. On SIGKILL/OOM/power-loss the §0.14 lock file is left pointing at a dead pid / freed port; the next client either hangs or attaches to an unrelated process now on that port. On attach, liveness-check the pid and probe the port + auth token before trusting the file; if dead, atomically reclaim it. Write it temp-then-rename so a crash mid-write can’t corrupt it.
• Control surface + cross-tier diagnostics. Add gaia daemon status|stop|restart| logs — an always-on process needs a way to see/stop/recover it (gaia kill is too blunt; it also kills the clock). Give each sidecar a log file under its state dir, stamp run_id (§0.1) into every daemon/sidecar/relay log line, and extend gaia diagnostics to gather daemon + all sidecar logs correlated by run_id — so a run spanning UI→daemon→sidecar→Lemonade is reconstructable and third-party-agent bug reports are actionable, not “it froze.”
• Daemon↔client version skew on update. §0.15 negotiates the agent contract; the UI/CLI↔daemon boundary is uncovered. An app update replaces the CLI/UI while the old daemon keeps running, so the new client attaches to a stale host API. Version the host API; on attach, mismatch the client can’t speak → drain in-flight runs and cleanly restart the daemon into the new binary. Decide where the daemon binary ships (core wheel vs hub) so “update the daemon” has an owner.

​

0.26 On-disk state layout & update survival

​

0.27 Relationship to sibling plans (supersedes / depends / reconcile)

• connectors.mdx + email-sidecar-agent-ui.md — “the sidecar reads grants.json directly” is superseded by §0.6. Those docs bundle gaia.connectors into the freeze so the binary can read the grant + keyring; v2 inverts this — the host owns consent+refresh and forwards a short-lived access token to the sidecar via /v1/connections; sidecars never touch the refresh token or the ledger. A sidecar reading grants.json is exactly the cross-process-writer race §0.6 removes. (Most material — it changes what the email cutover builds.)
• security-model.mdx “localhost-only is the trust boundary; CLI+UI are TRUSTED” is superseded by §0.11/§0.24. v2 states loopback is not an auth boundary and mandates the three auth legs + per-agent authorization + egress containment. An implementer reading only security-model.mdx would wrongly conclude localhost binding suffices.
• email-sidecar-agent-ui.md decision 4 (fixed-call forwarding only, no /query) is superseded — v2 makes /query (SSE) the primary UI chat surface (§0.10 step 1). PR #1910’s EmailProxyAgent is already flagged superseded (§0 header); the standalone doc needs a top banner pointing here.
• email-sidecar-agent-ui-implementation.md “the UI backend spawns/owns the sidecar; EmailSidecarManager lives in src/gaia/ui/” is the interim (email-first, §0.10 step 1) shape, not the v2 end state. In v2 the supervisor relocates into the headless daemon (§0.0/§0.3/§0.14); the UI and CLI only attach.

• autonomy-engine.mdx — the v2 host daemon IS the Autonomy Engine’s always-on background service. Both independently define an always-on process with a cron clock; they are the same process (recommended). The shipped in-UI /api/schedules router (schedules.py, #550) moves per §0.9: the clock + trigger registry are host/daemon, the job executes in the owning sidecar spawned at fire time (§0.22). Self-scheduling writes to the host clock, never sidecar-local state.
• MCP server ownership (connectors.mdx mirrors servers for an in-process MCPClient). v2 split: the host owns the MCP server registry/config (user-configured, shared — §0.9); a sidecar spawns/manages its own client connections to the servers its agent needs. The host does not proxy MCP traffic; it owns the config the sidecar reads (per-agent scoped, §0.11).
• setup-wizard.mdx model download vs §0.5 per-agent install-time provisioning. Two model-download triggers exist (wizard first-run, profile-level; §0.5 per-agent at install). Decide ownership + sequencing: the wizard handles the base profile model at first run; §0.5 handles additional per-agent models at install, both pulling through the host broker (§0.12); daemon auto-start (§0.25) must compose with wizard first-run detection so they don’t both fight over setup-state.json.

• agent-ui-hub-publish.mdx Part 2 “in-app Agent Hub + dynamic install” IS §0.5. Same work — cross-link them. Its multi-component model also resolves “is the UI an agent?” cleanly: the UI is an app, not an installable agent. Two requirements v2 layers on top that the hub docs must carry: install provisions the model, not just the binary (§0.5 cold-start), and sign the lock/catalog + anti-rollback (§0.24) — the hub docs specify SHA-256 + tiers, but SHA is integrity, not authenticity.

​

0.28 The agent manifest — the load-bearing artifact (was undefined)

• The lock = binary integrity (exists). SHA/signature-verified (§0.24).
• The manifest = capability + policy declaration (new; define it). It drives authorization decisions (egress, scopes, tier), so it must ride inside the §0.24 signature envelope — the recommended shape is a separate manifest.json referenced by digest from the signed lock, so its security fields can’t be swapped independently of the binary. A manifest outside the signed envelope is forgeable and worthless for authorization.

​

0.29 Custody store consistency model (concurrent daemon + N sidecars)

• Single physical writer. All custody writes go through the daemon; sidecars request writes via /host/v1/*, they never touch the files. This extends §0.6’s single-writer rationale to memory, RAG, sessions, and audit — so N sidecars writing “shared user-memory rows” (§0.20) still resolve to one physical writer.
• Serialized audit appends (hard requirement of §0.24’s hash-chain). §0.24’s tamper-evident chain needs a strict total order — “each entry seals the prior.” N sidecars POST /host/v1/audit concurrently would both seal entry k and fork the chain, which the gap-detector reads as tampering. The daemon must serialize appends behind a single append queue, or the tamper-evidence guarantee is undefined.
• Storage engine. The current UI stores are SQLite (single-writer; SQLITE_BUSY under concurrent writers). Specify WAL + busy_timeout (or a host-side write queue) and RAG read-during-index isolation, or “host stores it, sidecars query it” hits lock contention the first time two agents write.

​

0.30 Identifier catalog

• session id — host-minted, host-owned (§0.9); it is the authorization key the callback verifies (/host/v1/sessions/{id}, §0.11), so leaving its mint point + per-client-vs-per-session semantics undefined (the open §0.9 focus decision) leaves a security check keyed off an undefined id. Resolve with §0.9.
• action_id — mint on every consequential action; it’s the handle the §0.19 audit entry and §0.20 uninstall-scoping (“withdraw/revoke this action”) both need, and it appears nowhere in §0 today. Stamp it into the audit record.
• batch_id — batch-archive/undo handle (email spec); define its mint + lifetime.
• per-spawn secret vs client-auth token — already cleanly separated (§0.11/§0.24); no gap.

​

0.31 The /host/v1/* callback API (the reverse contract, specified)

​

0.32 Multi-agent orchestration (the headline “complex workflow automation”)

• An orchestrator is itself an agent (a sidecar), not host logic. Keeping the host thin (§0.0), the orchestrator is a first-class agent whose /query plans a multi-step, multi-agent workflow and invokes other agents. The host stays the router/custodian; the reasoning lives in the orchestrator sidecar.
• Agent-to-agent calls go through the host, never sidecar-to-sidecar directly. The host mediates so every hop is authorized (§0.11 per-agent scope), audited (§0.19 action_id), broker-leased (§0.12), and taint-tracked — a direct sidecar→sidecar mesh would bypass all four. Add a host route (POST /host/v1/agents/{id}/invoke, client-auth + orchestrator-scoped) so the orchestrator reaches sub-agents under the same controls as any client.
• Taint + confirmation are the safety backbone (§0.24 / §0.4). Untrusted content read by agent A (a mailbox body) that becomes agent B’s input is marked tainted so B treats it as data, not instructions (§0.24 cross-agent injection), and every destructive cross-agent step keeps the confirmation gate (§0.4 approve-what-you-saw) — the orchestrator cannot launder an injected instruction into an un-approved send.
• Cost is real and must be legible. A cross-agent workflow is N agent loops + M model switches on one slot (§0.12) — the orchestrator’s own loop plus each sub-agent’s. Surface it as streamed status (which agent is working) and hold interactive priority so a foreground orchestration preempts background jobs (§0.12).
• Sequencing: v1 = explicit picker (§0.18); the orchestrator is a later, first-class agent built on the same contract — not a special host mode. This keeps “the UI is thin, the sidecar reasons” intact even for multi-agent flows.

• Delegated (host present): full A2A as above — invoke through the host, which authorizes + audits + leases the slot + carries taint on every hop.
• Embedded, single agent: no A2A — one agent, no peers, no coordinator needed.
• Embedded, multi-agent (a vendor runs several sidecars): needs a coordinator — run the GAIA host or implement its invoke+custody+broker interfaces. Direct sidecar→sidecar is possible but loses the guarantees (taint doesn’t travel → §0.24 injection risk; no unified audit; the two agents contend for the model slot → the §0.12 eviction bug). Fine for trusted simple chaining; unsafe as a general pattern.
• Ephemeral: A2A is caller-orchestrated call-chaining — the caller invokes A, passes its output into B’s request; nothing shared or persisted; the caller owns all coordination.

​

0.33 The GAIA REST API agent server exposes /query too (src/gaia/api/)

• Add POST /v1/<agent>/query (SSE) to the API server, the same contract (§0.1/§0.2) — so a REST consumer gets tool-calling, multi-step workflows, and the typed event stream, not only OpenAI-style chat. One agent-loop implementation: the API server proxies to the sidecar (via the daemon/broker, §0.3/§0.12), exactly as the UI and CLI do — it does not run its own in-process loop.
• Relationship to /v1/chat/completions. Keep both: chat-completions is the OpenAI-SDK-compatible surface (drop-in for existing tooling); /query is the richer agentic superset (SSE tool events + needs_confirmation + workflows). Consumers choose by need; neither is removed.
• Supersedes the API server’s in-process email mount. openai_server.py:143 mounts gaia_agent_email.api_routes in-process — the same in-process pattern the UI cutover removed (this surface was explicitly left out of PR #1910’s scope). Under v2 it likewise becomes sidecar-backed (proxy to the email sidecar), so core stays lightweight and there is one email implementation.
• Auth is stricter here than the loopback daemon (🔒). gaia api is a network-exposed surface (and reachable remotely via the tunnel), unlike the daemon’s loopback custody API — so /query on it must sit behind the API server’s API-key auth and still enforce the §0.4 confirmation gate + §0.24 containment on every destructive step. Do not inherit the loopback trust assumptions of §0.11 here.

​

0.34 Autonomy readiness — infrastructure-ready, policy-hostile

1. Confirmation assumes a watcher. §0.4’s “approve-what-you-saw” pauses the SSE stream for synchronous approval — but unattended there’s nobody to approve. Full autonomy needs a policy / pre-authorization model: the user pre-grants categories of action (e.g. “auto-archive promotions,” “auto-decline conflicting invites”) with undo + audit, replacing per-action approval when unattended. This is the central missing piece.
2. Cron-only triggering (§0.22). Autonomy is largely event-driven (“urgent mail arrived → act”); needs a trigger/event bus + mailbox-watch, not just a clock.
3. No long-lived goals / self-initiation. The contract is request → /query → done; autonomy means the agent initiates from standing objectives it tracks over time (goals.py is noted moving to a sidecar, §0.9, but autonomous goal-pursuit is undesigned).
4. No async escalation. An unattended agent that hits uncertainty should notify and resume later when the user answers — but §0.4 confirmation is synchronous. Needs a notify-and-resume path (push notification → deferred approval → continue).
5. No autonomy levels. There are third-party trust tiers (§0.24) but no graduated autonomy (observe → suggest → act-with-undo → act-freely) governing what an agent may do unattended, per capability.
6. Ephemeral vs. continuous. §0.13 reaps idle sidecars; a monitoring agent doesn’t fit “spawn-at-fire, reap-after.” Pinned-resident (§0.22) is the exception, not a first-class monitoring model.

• broker priority (§0.12) as its guardrails. It also reconciles with autonomy-engine.mdx (§0.27): that engine is this layer, hosted in the daemon. Sequence it after the human-in-the-loop v1 — the v1 confirmation model is the safe default, and the autonomy layer is what lets the user progressively hand off.

​

0.35 Structural refinements (from the architecture review)

1. Sidecar independence is a two-tier contract, not a flat claim. The sidecar owns no durable state (§0.9), so custody-backed /query needs a host. Make it honest: publish /host/v1/* (§0.31) as a first-class, third-party-implementable custody interface (a third party can bring their own custodian), and formalize the bare-integrator degraded tier (§0.6) as a shipped, tested product with a capability matrix — works without a host: fixed-function + stateless /query; needs a host: memory, RAG, sessions/transcript, audit. Converts the biggest hidden coupling into an explicit contract tier. (Reflected in the mdx overview.)
2. Naming: never call the daemon “thin.” “The host” = the custodian daemon, the most responsibility-dense process; “thin” is the UI only. (Applied across the docs.)
3. Assert the daemon’s internal module seams. One process is right; one module would rot into a monolith. Commit — even inside the process — to separated custody-store / broker / supervisor / proxy-router / clock modules with defined interfaces. Mark the broker (§0.12) as the one designed to be later-extractable to its own process: it arbitrates a hardware resource contended by non-agents (host RAG embedder, voice, SD), a different axis + lifecycle from data custody.
4. Slogan: “one agent contract, many front-doors; two control-plane contracts behind it” (the callback §0.31 and the client↔daemon §0.11/§0.25) — so B/C get first-class versioning/auth, not footnote status. (Reflected in the mdx.)
5. Trim three pieces of premature hardening to “third-party gate, not v1 build” (all additive later; acceptable under the first-party single-user threat model):
   • Audit: v1 = a plain append-only host-owned log (the single writer already gives order); the §0.24 hash-chain + §0.29 global single-append-queue are the third-party addition, not day-one.
   • Egress: v1 = the proxy-enforced allowlist (§0.24); the full network- namespace sandbox is research-grade/per-OS — defer.
   • Broker: v1 = priority queueing (foreground jumps the queue); defer mid-run preemption (§0.12) — you can’t cleanly pause a llama-server generation.
6. Make the render boundary real (layering fix). The “thin UI” has a hidden compile-time dependency on first-party agents’ card set (§0.2/§0.15). Ship a small fixed set of generic render primitives (table / key-value / list / image / diff) once; agents default to them, a bespoke component is the first-party exception — so “the thin UI renders any agent” is true rather than an ever-growing first-party component map.

​

0.36 Third-party integration ergonomics (a first-class design goal)

1. Custody (the §0.35 #1 seam). The sidecar owns no durable state (§0.9), so the rich agentic experience (memory/RAG/audit-backed /query) needs a host. Make the tier a product: publish /host/v1/* (§0.31) as a first-class, third-party- implementable interface and ship the bare-integrator tier (§0.6) as a tested product with a published capability matrix (below). Then “run our custodian,” “bring your own custodian,” and “degraded standalone” are three documented choices, not a cliff.
2. The LLM backend. The sidecar needs a reachable model backend (Lemonade), and some agents (email) force local-only by policy — so the artifact is “agent + LLM backend,” not one binary. Make the backend an explicit, documented dependency with a configurable base_url, and state per-agent whether a cloud/remote backend is allowed (email = local-only by design; others may differ). A vendor must know “you also provide the model host” up front.

​

0.37 Pluggable custody — one sidecar, both experiences (rich and stateless)

• Embedded (self-custody).
  • Pros: fully self-contained rich agent — one binary, no external custody dep; trivially its own single writer (no coordination); works offline/air-gapped; simplest deployment; ideal for a third-party vendor who wants “rich, standalone.”
  • Cons: state is siloed per agent — no shared cross-agent user memory (each agent has its own), so a multi-agent setup fragments the user’s memory + duplicates shared data (e.g. grants); the sidecar owns store lifecycle (schema, migration, backup, encryption); audit is local + per-agent (uninstall can lose it).
• Delegated (host custody).
  • Pros: one coherent user — memory/RAG/audit unified across every agent; one place to encrypt-at-rest, back up, govern, and enforce uninstall lifecycle; the sidecar stays truly light (no store in the binary); enables cross-agent orchestration over shared memory (§0.32).
  • Cons: requires a running host (the coupling — custody-backed features don’t work standalone); a network hop + auth on every memory/RAG call; the host is a single point of failure for all agents’ custody; carries the full cross-process contract cost (/host/v1/* + auth legs + versioning).
• Ephemeral (stateless).
  • Pros: nothing on disk — simplest, most portable, most secure at rest; horizontally scalable (any instance serves any request); no migration/backup/encryption concerns; easiest to test; ideal for one-shot/scriptable/serverless integration.
  • Cons: no memory or learning — no personalization, follow-up tracking, or cross-session context; the caller must pass all context every request (bigger payloads, caller owns state); no persisted audit trail; it’s a function, not an assistant.

• One binary does both: a vendor gets a rich, self-contained agent or plugs in their own custody host; GAIA gets shared multi-agent custody. §0.36’s “high-effort” tier becomes low-effort (embedded is the default).
• Footprint stays light: embedded custody is storage (SQLite + index files), not ML libs — the embedder compute still runs through the external model backend, so §0.21 is untouched.
• /host/v1/* is now doubly justified: it’s both the delegated-mode wire protocol and the third-party-implementable interface (§0.35 #1) — the embedded adapter is simply GAIA’s own reference implementation of it.
• Cost: mode migration (embedded→hosted / export) reuses the §0.10 machinery; the embedded and delegated adapters must pass the same conformance tests (§0.17) so behavior is identical across backends.

​

0.38 Practical memory / resource footprint

• The model dominates; the broker (§0.12) is the primary memory governor. A 4B GGUF is ~2–4 GB resident. The broker’s single model slot keeps ~one model resident regardless of agent count — without it, N agents wanting models = N models = OOM. So §0.12 is a memory-protection mechanism, not only a correctness one.
• Per-sidecar overhead (~50–150 MB RSS each) is bounded by reaping. N installed agents ≠ N resident processes — lazy-spawn + the idle-reaper + the live-sidecar cap (§0.13) keep only active agents in memory. Second governor.
• Custody mode changes the multiplier: embedded duplicates the store and the RAG index in RAM per agent (heaviest multi-agent); delegated holds shared custody + one RAG index in the host (leanest multi-agent; host RSS grows instead); ephemeral holds no store (lightest). The RAG/FAISS index (tens–hundreds of MB in RAM per query) is the sneaky duplicator — embedded = N copies, delegated = 1.
• Model switching is churn, not resident bloat — different-model agents evict+reload (~GB I/O + a stall), serialized by the broker; hot-model affinity (§0.12) reduces thrash.
• Steady state ≈ one model + the daemon + the active sidecars + (Electron UI ~300 MB, only when open) — not “sum of everything installed.”

​

0.39 Custody modes at a glance — consolidated comparison

​

0.40 Hybrid custody — per-store provider (the embedded↔delegated spectrum)

• local app state uses); and weaker coupling / graceful degradation — an agent needs the host only for its delegated stores, so a host outage degrades shared features while embedded ones keep working (partial-host becomes a real, tolerable state).

​

0.41 Separation of concerns — one clean authorization plane, not one god-grant

• §0.32 A2A-routing really are one primitive (a scoped, host-checked grant). Keep them one.

• The grant checks the tool boundary, not the reasoning. In an LLM loop the tools are the capabilities but the model chooses which to call over tainted context — a static “may A call tool T” cannot judge “should this call, with these args, happen.” That judgment is the confirmation gate + treat-input-as-data (the confused-deputy answer), not the grant. Don’t claim the grant secures the reasoning.
• Revocation (simple, concrete). Grants are mutable host state keyed by agent-id; the mediator re-reads the grant per mediated call (cheap, in-memory) — revoke = update the grant. An in-flight autonomous batch re-checks per op (already mediated); revoking A stops further A2A invokes while B’s in-flight call completes under B’s own grant. (No new machinery — it’s a lookup the mediator already does.)
• Embedded has NO mediation plane — say so. Embedded single-agent is a trust boundary of one — a storage adapter (§0.37), not a policy-enforcement point. The mediation plane exists only where multiple mutually-distrusting agents share a host (delegated). Drop the “embedded = a minimal host that enforces” claim (it would ship a stub PEP that enforces nothing). So the modes are not “one architecture, varying host” — they’re one contract, with a real PEP only in the multi-agent (delegated) case.
• Audit actions, not reads. §0.24/§0.29’s serialized hash-chain covers consequential actions (low-frequency, tamper-evident) — do not put a machine-wide append-lock on every store read. Audit is host-if-present, not part of the per-store hybrid (§0.40) freedom.
• Taint is a defense-in-depth hint, not a guarantee — it dies through LLM summarization (§0.42) and is non-durable across a daemon restart. The real backstops are isolation + egress + confirmation + audit. Treat cross-agent/untrusted input as data regardless of taint.

​

0.42 Prior art & learnings (OpenClaw · Hermes · MCP/A2A)

• Coordinator + one agent core. OpenClaw’s Gateway (one port multiplexing sessions, tool dispatch, routing, orchestration) and Hermes’s single AIAgent core reused by CLI / gateway / ACP server / cron are the same pattern as our custodian daemon + one-contract-many-front-doors (§0.33). Independent convergence → sound.
• OpenAI-compatible server (Hermes exposes /v1/chat/completions) = our §0.33.
• Shared-or-scoped skills (OpenClaw scopes skills global↔per-agent) = our shared/private custody axis (§0.40).
• Persistent memory is table-stakes — Hermes (“grows with you,” a GraphRAG “second brain”) validates our custody/memory direction; consider GraphRAG for RAG.

1. Align with the MCP + A2A two-layer standard — don’t invent. MCP (Anthropic’s tool layer; ~97M monthly SDK downloads; adopted by Anthropic/OpenAI/Google/Microsoft/ Amazon) is the vertical tool protocol → expose our fixed-function endpoints as MCP tools (the email agent already ships mcp_server.py — make it first-class). Google’s A2A protocol is the horizontal agent-coordination layer → align §0.32 A2A with A2A Agent Cards (capability discovery), signed payloads, and delegation control instead of a proprietary invoke route. Payoff: any MCP/A2A client consumes our agents out of the box → §0.36 third-party digestibility jumps.
2. Process isolation is our security advantage — lean in. OpenClaw’s documented core flaw — “a single compromised skill inherits ALL permissions” (plus multiple arXiv injection/trojan papers) — is a shared-runtime problem. Our per-agent process- isolated sidecars + per-agent grants contain a compromised agent to its own grant. Strictly better isolation; position it as a differentiator.
3. Use a permission cascade to blunt consent fatigue. OpenClaw’s global→provider→ agent→session→sandbox cascade sets sensible defaults at each level so the user consents only to deviations. Adopt trust-tier defaults → user overrides (§0.24), far better than per-capability prompts.
4. Nobody ships formal information-flow control — this confirms the §0.41 correction. MCP’s own security guidance is “read-only for untrusted agents, scoped permissions, network restrictions” — i.e. the same pragmatic stack (isolation + scopes + egress
   • human approval + audit) we landed on. Don’t build formal IFC.
5. A2A protocol’s “prevents unauthorized delegation” is the confused-deputy answer — adopt its delegation-control + payload signing for our A2A hops.

​

1. Current GAIA SDK Capability Inventory

​

1.1 Agents

​

1.2 ChatAgent Tools (Current — What the Agent UI Uses)

​

1.3 CodeAgent Tools (Available in SDK, NOT in Agent UI)

​

1.4 Other SDK Capabilities (Not Exposed to Any Agent)

​

2. Gap Analysis: Agent UI Agent vs. Modern PC Agent Expectations

​

2.1 Capabilities Users Expect Today

​

2.2 What Can Be Added to ChatAgent NOW (Reusing Existing SDK)

​

2.3 What Requires New Development

​

3. Implementation Plan

​

Phase 1: Quick Wins — Wire Existing SDK into ChatAgent (1-2 weeks)

class ChatAgent(
    Agent,
    RAGToolsMixin,         # Existing: document Q&A
    FileToolsMixin,        # Existing: file watching
    ShellToolsMixin,       # Existing: shell commands
    FileSearchToolsMixin,  # Existing: file search
    FileIOToolsMixin,      # NEW: read/write/edit files
    ProjectManagementMixin,# NEW: list_files, create_project
    ExternalToolsMixin,    # NEW: web search, doc search
    TestingMixin,          # NEW: execute Python, run tests
):

​

Phase 2: Vision & Media (2-3 weeks)

​

Phase 3: Web & System (2-3 weeks)

​

Phase 4: Computer Use (4-6 weeks, separate milestone)

​

Phase 5: Audio/Voice (2-3 weeks)

​

4. Cross-Platform Requirements

​

5. Security Considerations

​

6. Issue Tracker

​

Already Created (Milestone #15)

​

To Create (Phase 1-5)

​

7. MCP Server Integration

​

7.1 Current MCP Infrastructure in GAIA

• MCPClientMixin (gaia/mcp/mixin.py) — Any agent can connect to MCP servers and auto-register their tools
• MCPClientManager — Manages multiple MCP server connections
• Config file — ~/.gaia/mcp_servers.json for persistent server configuration
• MCPAgent base class — agents/base/mcp_agent.py
• MCP Bridge — gaia/mcp/mcp_bridge.py exposes GAIA as an MCP server to external tools
• Existing integrations — Docker MCP, Blender MCP already implemented

​

7.2 Most Popular MCP Servers (2026 Ecosystem)

​

Tier 1 — Essential (High demand, directly useful for Agent UI)

​

Tier 2 — High Value (Popular integrations users commonly request)

​

Tier 3 — Windows Desktop Automation (Key for “Computer Use”)

​

Tier 4 — Microsoft Ecosystem (Enterprise)

​

7.3 Agent UI MCP Integration Design

​

A) MCP Server Manager Panel (Settings)

1. Browse/add popular servers from a curated list (Tier 1-2 above)
2. Add custom servers by providing command + args + env config
3. Enable/disable servers per session or globally
4. View connected server status (connected, tools available, errors)
5. Configure server credentials (API keys, tokens) with secure storage

​

B) Backend API Endpoints

​

C) ChatAgent MCP Integration

class ChatAgent(
    Agent,
    MCPClientMixin,        # NEW: MCP server connectivity
    RAGToolsMixin,
    FileToolsMixin,
    ShellToolsMixin,
    FileSearchToolsMixin,
    # ... other mixins
):

1. Calls agent.connect_mcp_server(name, config)
2. Tools from the MCP server are auto-registered in the agent’s tool registry
3. The agent can now use those tools in its planning/execution
4. Tools appear in the Capabilities panel (#440)

​

D) Curated Server Catalog

{
  "catalog": [
    {
      "name": "filesystem",
      "display_name": "File System",
      "description": "Secure file read/write/search with configurable access",
      "category": "system",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed"],
      "requires_config": ["allowed_directories"],
      "tier": 1
    },
    {
      "name": "github",
      "display_name": "GitHub",
      "description": "Repos, PRs, issues, workflows",
      "category": "dev-tools",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {"GITHUB_TOKEN": ""},
      "requires_config": ["GITHUB_TOKEN"],
      "tier": 1
    },
    {
      "name": "playwright",
      "display_name": "Browser (Playwright)",
      "description": "Web browsing and interaction via accessibility snapshots",
      "category": "browser",
      "command": "npx",
      "args": ["-y", "@anthropic/mcp-playwright"],
      "tier": 1
    }
  ]
}

​

7.4 MCP Issues to Create

​

8. Dependencies

Phase 1 (Quick Wins — Existing SDK Mixins)
  └── No external dependencies (reuse existing SDK)

Phase 1-MCP (MCP Server Integration)
  ├── MCPClientMixin (already exists in SDK)
  ├── Node.js/npx (for npm-based MCP servers)
  └── MCP server packages installed on demand

Phase 2 (Vision & Media)
  ├── Lemonade Server with VLM model loaded (Qwen3-VL-4B)
  ├── PIL/Pillow (already in deps)
  └── mss (new dep for cross-platform screenshots)

Phase 3 (Web & System)
  ├── httpx (already in deps)
  ├── beautifulsoup4 (new dep)
  ├── pyperclip (new dep)
  └── plyer (new dep for notifications)

Phase 4 (Computer Use)
  ├── Phase 1 (guardrails MUST be done first)
  ├── Phase 2 (VLM for screen understanding)
  ├── pyautogui (new dep)
  ├── pywinauto (Windows, new dep)
  ├── Playwright (optional, for browser automation)
  └── OR: Windows Desktop MCP servers (external, via MCP)

Phase 5 (Audio/Voice)
  ├── Whisper ASR model loaded in Lemonade
  ├── Kokoro TTS model loaded in Lemonade
  └── Browser MediaRecorder API support

​

9. Critical Capabilities Coverage Matrix

​

9.1 Browser Control — Needs Priority Bump

BEFORE:  Phase 4g (week 8+) — build native Playwright integration
AFTER:   Phase 1-MCP (week 1) — enable Playwright MCP server
         Phase 4g (later) — build native integration only if MCP insufficient

​

9.2 Web Search — Needs Free Fallback

1. Brave Search MCP (free, no API key for basic usage)
2. Perplexity (if API key available, via ExternalToolsMixin)
3. Fetch MCP (raw URL fetch + markdown conversion as last resort)

​

9.3 Shell Commands — Write Capability Gap

• mkdir — create directories
• cp/mv — copy/move files
• pip install — install packages
• npm install — install node packages

1. FileIOToolsMixin for file write/create/edit (Phase 1a)
2. CLIToolsMixin for broader command execution (Phase 1, with guardrails)
3. Guardrails (#438) to confirm dangerous operations

​

9.4 Guardrails — The Critical Prerequisite

Week 1: Guardrails framework (#438) + read-only mixins (ProjectManagementMixin)
Week 2: FileIOToolsMixin + ExternalToolsMixin (with guardrails active)
Week 3: CLIToolsMixin + TestingMixin (with guardrails active)

​

9.5 Email & Calendar — New Capability Needed

​

MCP Servers Available

​

User Workflows (Email)

​

Priority

​

9.6 Application Control (CUA) & Popular App Demos

​

MCP Servers for Application Control

​

CUA (Computer Use Agent) Strategy

1. MCP-first (preferred): Use app-specific MCP servers for structured, reliable control. Spotify MCP is better than clicking the Spotify UI because it’s API-driven, reliable, and doesn’t break when the UI changes.
2. Vision + automation (fallback): For apps without MCP servers, use:
   • Screenshot → VLM (identify UI elements) → pyautogui (click/type)
   • This is Phase 4 in the plan and requires VLM + guardrails

​

Demo Workflows

​

10. Detailed Plan Critique & Issues Found

​

10.1 CRITICAL: FileIOToolsMixin Has Hidden Dependency

class FileIOToolsMixin:
    """...
    NOTE: This mixin expects the agent to also have ValidationAndParsingMixin
    for _validate_python_syntax() and _parse_python_code() methods.
    """

1. Add ValidationAndParsingMixin to ChatAgent (drags in CodeSymbol, ParsedCode models, validator classes)
2. Refactor FileIOToolsMixin to make Python validation optional (try/except around _validate_python_syntax)
3. Create a lightweight ChatFileIOToolsMixin that strips out Python-specific features

# In read_file, for .py files:
if hasattr(self, '_validate_python_syntax'):
    validation = self._validate_python_syntax(content)
    result["is_valid"] = validation["is_valid"]
else:
    result["file_type"] = "python"  # still tag it, just skip validation

​

10.2 CRITICAL: _TOOL_REGISTRY Is Global — Tool Count Explosion

• ChatAgent: ~13 tools (12 @tool decorators across 3 files)
• CodeAgent: ~57 tools (69 @tool decorators across 12 files, minus register functions)

• Reduced context for actual conversation history
• LLM confusion from too many tool choices (decision paralysis)
• Slower inference (more tokens to process)

1. Lazy tool loading: Only register tools when their category is needed (e.g., don’t load Prisma tools if user isn’t doing database work)
2. Tool description compression: Use 1-2 sentence descriptions in prompts, not full docstrings
3. Tool categories: Group tools and let the LLM request a category expansion
4. Per-session tool selection: Let users enable/disable tool categories from the UI (ties into #440 Agent Capabilities Discovery)

​

10.3 HIGH: ExternalToolsMixin — Silent Failure Risk

• get_context7_service() — requires npx on PATH (Node.js installed)
• get_perplexity_service() — requires PERPLEXITY_API_KEY env var

def register_external_tools(self):
    if shutil.which("npx"):
        # register search_documentation
    if os.environ.get("PERPLEXITY_API_KEY"):
        # register search_web

​

10.4 MEDIUM: MCP-vs-Native Build/Buy Confusion

• Duplicate tools in registry (LLM sees read_file AND filesystem__read_file)
• Conflicting behaviors (different error formats, different security models)
• Maintenance burden

• Native tools for core, always-available capabilities (file I/O, shell)
• MCP servers for external integrations and optional capabilities (GitHub, Spotify, email)
• MCP preferred when the MCP server is more capable (Playwright MCP > building our own)
• Never both for the same capability in the same session

​

10.5 MEDIUM: Missing Effort Estimates & Timeline Reality

• Guardrails framework (#438) — design + implement + test = 1 week minimum alone
• FileIOToolsMixin refactoring (§10.1) — 2-3 days
• MCP Server Manager UI — new Settings tab + API endpoints + state management = 1 week
• Testing all new tools on Windows + Linux = 3-5 days

Phase 1 actual: 3-4 weeks (not 1-2)
Phase 1-MCP actual: 2-3 weeks (not concurrent with Phase 1)
Phase 2 actual: 3-4 weeks (VLM model loading is non-trivial)

​

10.6 LOW: Cross-Platform Testing Gaps

• CI/CD: No GitHub Actions matrix for Windows/Linux/macOS testing
• pyautogui on headless: Won’t work in CI without virtual display
• macOS permissions: Screenshot, accessibility, and automation all require explicit System Preferences permissions that can’t be automated

​

11. User Workflow Validation Tests

​

11.1 File Operations Workflows

​

11.2 Web & Search Workflows

​

11.3 Document Analysis Workflows

​

11.4 Shell & System Workflows

​

11.5 Email & Calendar Workflows

​

11.6 Browser & App Control Workflows

​

11.7 Guardrails Validation Workflows

​

12. Updated MCP Server Catalog (Complete)

​

Tier 2+ — Communication & Productivity (NEW)

​

13. New SDK Capabilities: MCP Auto-Discovery & SKILL.md Support

​

13.1 MCP Server Auto-Discovery & Installation

User: "Check my email for anything urgent"
Agent: I don't have email access yet. I found these MCP servers that can help:
       1. Gmail MCP (gmail-mcp-server) — Gmail access
       2. Outlook MCP (outlook-mcp-server) — Outlook/Microsoft 365
       3. Nylas MCP (nylas-mcp-server) — Multi-provider (Gmail + Outlook + more)
       Would you like me to install one?
User: "Install Gmail MCP"
Agent: [installs via npx, prompts for OAuth/credentials, connects]
       Gmail MCP is now connected. You have 3 urgent emails...

{
  "capabilities": {
    "email": ["gmail-mcp-server", "outlook-mcp-server", "nylas-mcp-server"],
    "calendar": ["google-calendar-mcp", "outlook-calendar-mcp"],
    "browser": ["@anthropic/mcp-playwright"],
    "web_search": ["@anthropic/mcp-brave-search"],
    "music": ["spotify-mcp-server"],
    "notes": ["obsidian-mcp-server"],
    "tasks": ["todoist-mcp-server"],
    "code": ["@modelcontextprotocol/server-github", "@modelcontextprotocol/server-git"],
    "files": ["@modelcontextprotocol/server-filesystem"],
    "database": ["@modelcontextprotocol/server-postgres"]
  }
}

​

13.2 Anthropic SKILL.md Support for GAIA Agent SDK

# Deploy GAIA Application

## Description
Steps to deploy the GAIA application to production.

## Prerequisites
- Docker installed
- Access to container registry
- `.env.production` file configured

## Steps
1. Run tests: `pytest tests/ -x`
2. Build Docker image: `docker build -t gaia:latest .`
3. Push to registry: `docker push registry.example.com/gaia:latest`
4. Deploy: `kubectl apply -f k8s/deployment.yaml`

## Learned
- Always run tests before building (learned 2026-03-01)
- Use `--no-cache` flag if dependencies changed (learned 2026-03-05)

• Skills can reference RAG documents (“See indexed doc: architecture.pdf”)
• Skills can reference MCP servers (“Requires: gmail-mcp-server”)
• Skills can include tool sequences that the agent replays
• Skills directory is auto-indexed by RAG for semantic search

​

14. Summary: Recommended Priority Order (Revised, Split by Milestone)

​

Milestone A — Agent UI: Wire Existing SDK (Weeks 1-6)

IMMEDIATE (This branch — kalin/agent-ui)
  ├── ✅ Windows shell compatibility fix (done)
  ├── ✅ Sidebar minimize + resize (done)
  └── ✅ Milestone + issues created (#438-#442)

WEEK 1-2: Foundation + MCP Framework
  ├── Add MCPClientMixin to ChatAgent
  ├── MCP Server Manager UI panel in Settings
  ├── Curated MCP server catalog (Tier 1: Playwright, Brave, Fetch, Filesystem, Git)
  ├── Refactor FileIOToolsMixin for graceful degradation (§10.1)
  ├── Conditional tool registration for ExternalToolsMixin (§10.3)
  └── Capability-to-MCP mapper (curated catalog — §13.1)

WEEK 3-4: Wire Existing Mixins + MCP Tier 1
  ├── Add FileIOToolsMixin to ChatAgent (file read/write/edit)
  ├── Add ProjectManagementMixin (list_files)
  ├── Add ExternalToolsMixin (web search, conditional)
  ├── Enable Playwright MCP (browser control)
  ├── Enable Brave Search MCP (web search, free)
  ├── Enable Fetch MCP (URL content extraction)
  └── Agent capabilities discovery API (#440)

WEEK 5-6: MCP Tier 2 + Productivity
  ├── MCP Tier 2 servers: Gmail, Outlook, Calendar, Spotify, Obsidian
  ├── MCP install prompt UI ("Install MCP server?" card)
  ├── Skills UI panel in Settings (view/manage SKILL.md files)
  ├── Tool argument streaming (#441)
  └── Per-session MCP server enable/disable

​

Milestone B — GAIA Agent SDK: New Capabilities (Weeks 3-12+)

WEEK 3-5: Guardrails + Safety (PARALLEL with Milestone A)
  ├── Tool execution guardrails framework (#438) ← MUST BE FIRST
  │   ├── OutputHandler.confirm_tool_execution() API
  │   ├── SSE handler → frontend confirmation modal
  │   ├── threading.Event blocking pattern
  │   ├── Allow-list with localStorage persistence
  │   └── Risk classification (read/write/execute/destructive)
  ├── Cooperative execution cancellation (#439)
  └── Cross-platform shell compatibility (#442)

WEEK 5-7: Vision & Media
  ├── ScreenshotToolsMixin — cross-platform (PIL.ImageGrab, mss)
  ├── Wire VLMToolsMixin into ChatAgent (image analysis)
  ├── Image display in Agent UI (base64/file URL)
  ├── Screenshot → VLM → describe workflow
  └── Wire SDToolsMixin (image generation, optional)

WEEK 7-9: SDK Architecture
  ├── Tool categories + lazy loading (§10.2)
  ├── Tool description compression for prompts
  ├── MCP auto-discovery (search npm/GitHub for servers) — §13.1
  ├── SKILL.md support (load, save, search, format) — §13.2
  └── Skill-RAG integration (auto-index skills directory)

WEEK 9-12: Computer Use (CUA)
  ├── Windows Desktop MCP integration
  ├── Mouse/keyboard control (pyautogui) with mandatory guardrails
  ├── Window management (pywinauto/wmctrl)
  ├── VLM-based screen element detection
  └── CUA demo workflows (open apps, fill forms)

LATER: Audio/Voice
  ├── Voice input (Whisper ASR in Agent UI)
  ├── Voice output (Kokoro TTS in Agent UI)
  └── Continuous voice conversation mode

​

Milestone Dependency Map

Milestone A (UI/Wiring)          Milestone B (SDK)
═══════════════════════          ═════════════════
Week 1: MCPClientMixin
Week 2: MCP Server UI            Week 3: Guardrails (#438) ← blocks write tools
Week 3: FileIOToolsMixin ───────────────→ needs guardrails for write ops
Week 4: Playwright MCP            Week 4: Cancellation (#439)
Week 5: Email/Calendar MCP        Week 5: ScreenshotToolsMixin
Week 6: Skills UI ────────────────Week 6: SKILL.md SDK support
                                  Week 7: Tool categories
                                  Week 8: MCP auto-discovery
                                  Week 9-12: Computer Use