Browser Use — Scope

Status: Phases 1–2 shipped, with deviations — see What actually shipped before treating anything below as current design. Phase 0’s measurements are still owed; Phases 3–4 (native CDP driver, record/replay) are not started. See the guide for how to use what landed. Supersedes: the “no browser engine” half of Browser Tools (decision D1) Issues: #458 (browser tools), #460 (desktop CUA), #224 (CUA via MCP)

What’s broken

GAIA can read the open web but cannot use the web. fetch_page is requests + BeautifulSoup — one GET, no JavaScript, no cookies, no session. That is fine for docs and articles and worse than useless where the user actually needs help. Measured against the current fetcher:

Target	Result
`example.com`	142 chars of text — works
`react.dev` (SSR’d SPA)	8,343 chars — works
`x.com/amd`	2,279 chars, and all of it is `"Log in Sign up…"` — the page behind the wall is invisible

The failure mode is not JavaScript. Server-rendered SPAs extract fine. The failure is authentication — and secondarily bot gates and genuinely client-rendered content. Every shipped skill that watches a page (price-watch, source-watch) inherits that hole: point either one at a retailer that requires a session and it silently reads a login screen. Note what is not in scope. Google, Microsoft and GitHub already authenticate through connectors via OAuth, which is strictly better than driving a browser — real tokens, real scopes, no screen-scraping. Browser login is for the long tail with no API.

The thing to get right first: what actually costs time

The premise behind “Playwright is slow, let’s build a fast vision-driven one” does not survive contact with this repo’s own numbers. Per-step wall clock decomposes roughly like this:

Step component	Cost	Whose fault
CDP command (click, type, read)	1–10 ms	nobody
Page settle / network idle	0.1–2 s	the web’s — every driver pays it
Browser cold launch	1–3 s	amortizable — keep the browser alive
LLM decides the next action	2–15 s	this is the turn

The last row dominates by an order of magnitude, and it is measured, not assumed. From the flagship latency study: Gemma-4-E4B on the GPU profile prefills at 387 tok/s and generates at 34 tok/s, so every 1,000 prompt tokens costs ~2.6 seconds. Committed eval baselines put TTFT between 0.31 s and 2.97 s per call. The flagship already carries ~17,000 tokens of fixed prefill — 43.9 s — before the page is even described to it. Two conclusions follow, and they point away from the original instinct. Swapping Playwright for a hand-rolled driver saves milliseconds. Playwright’s real costs are its ~200 MB browser download and its cold launch, not its per-action latency. Those are install-time and session-time costs, not per-step ones. A vision-first loop is slower and less accurate, not faster. A screenshot is the most expensive possible way to describe a page to a model that charges 2.6 s per 1,000 tokens — an image costs on the order of 1,000–2,000 prefill tokens where a filtered accessibility tree costs a few hundred. Worse, the reliability argument cuts the same way: a 4B model predicting pixel coordinates is guessing, while a DOM ref is exact. Gemma-4-E4B in particular is not a GUI-grounding model; if vision is used at all it should be Qwen3-VL, which is trained for it. So: text-first, vision as a narrow fallback. Vision earns its place on <canvas> apps, image CAPTCHAs we decline to solve anyway, and “does this page look right” verification — not as the primary observation channel.

Where the real speedup actually is

Not in the driver. In not calling the model at all. A browser task is overwhelmingly repetitive — the same login, the same three clicks, the same table, every morning. Record a successful trajectory as a deterministic script and replay it; on the second run the LLM is consulted only when replay diverges from what it recorded. price-watch goes from ~8 LLM steps (~30 s) to zero (~2 s). That is the 10–100×, and it is a caching problem, not a driver problem. This has a natural home already: skills carry permissions: and tools_required: metadata, and skill_learning_tools.py already lets an agent write back what it learned under provenance constraints.

What actually shipped

This plan was written before the work; three things landed differently, and the sections below still describe the plan rather than the code.

Planned	Shipped	Why
Nine tools, incl. `browser_select`, `browser_scroll`, `browser_back`, `browser_screenshot`	Six — select folded into `browser_type`, scroll dropped (Playwright auto-scrolls on click), back and screenshot deferred	Bundles cap at six members, and each tool schema is a permanent prompt cost
Browser runs as a supervised daemon sidecar (D7)	In-process, owned by the agent, torn down on cleanup	The sidecar registry is spec-and-health-endpoint driven for agent packages, not a generic process supervisor; `MAX_LIVE_SIDECARS` is 3 with two already claimed
A `BrowserDriver` protocol behind which Playwright sits	`PlaywrightDriver` used directly	No second implementation yet; the protocol buys nothing until Phase 3

Phase 2’s origin-scoped skill grants (browser:interact:<domain>) did not land either. What did land is the confirmation gate: acting inside a session the browser context holds cookies for prompts every time, and is never offered an “always allow”.

Recommendation

Build the capability in this order. The driver is the swappable part; the agent loop is the hard part.

Put every driver behind one interface (~12 methods). Nothing above it knows which engine is underneath.
Ship Playwright first as the reference implementation — behind an optional [browser] extra, never a core dependency. It buys mature auto-waiting, a real selector engine, and storage_state persistence, which is weeks of work to reimplement badly. Use it to prove the loop.
Then replace it with a native CDP driver once the observation format and action vocabulary are frozen.

The case for step 3 being genuinely cheap, and why this is not premature:

Zero new Python dependencies. aiohttp is already in install_requires; CDP is JSON over one WebSocket.
Zero new browser download for desktop users. electron@^44 is already a build dependency of the Agent UI and the app ships packaged via electron-builder — Chromium is already on disk, and webContents.debugger is CDP. On Windows, Edge is preinstalled and takes --remote-debugging-port.
It buys the UX. A GAIA-owned window can be watched, paused and taken over by the user mid-task. That matters enormously for login, and Playwright’s window is not ours to render.

That last point is why this is not just NIH: the 200 MB download conflicts with GAIA’s edge positioning, and the login flow needs a window the user can grab. GAIA never types the password. The model does not see, store, or submit credentials — this is non-negotiable and it is also the only design that survives MFA. The flow mirrors what OAuth already does in connectors:

Agent hits a login wall and stops.
GAIA opens a visible browser window on the login page and hands the user the keyboard.
User authenticates however they like — password manager, passkey, MFA, SSO.
GAIA detects success, captures session state only (cookies + localStorage), and persists it encrypted, scoped to that origin.
Later runs restore the session; expiry sends the user back to step 2.

Where the session blob lives

Not in the keyring. This is a concrete constraint, not a preference: Windows Credential Manager caps entries at ~1280 characters, a cookie jar runs to tens of KB, and connectors/mcp_server.py writes secrets with a bare keyring.set_password — the chunking helpers that handle this (_kr_set/_kr_get in connectors/store.py) are private to that module. A storage-state blob down that path fails on Windows. Instead: key in the keyring, ciphertext on disk.

~/.gaia/browser/sessions/<origin-hash>.enc   0600, AES-GCM
keyring: gaia.connections / browser-session-key   (32 bytes)

cryptography is already a core dependency. Non-secret metadata (origin, captured-at, last-used, expiry) goes in a sibling 0600 JSON file so listing what is stored never touches plaintext. There is no prior art for this in connectors — ConnectorSpec.type is Literal["oauth_pkce", "mcp_server"], with no cookie/session type. Adding a third type is a possible follow-up; it should not block this.

Architecture

Planned, and not what shipped — the browser runs in-process. See What actually shipped.

The browser runs as a supervised sidecar, not in-process. It is a long-lived, crash-prone, memory-hungry process — exactly what src/gaia/daemon/sidecars/ exists to babysit, and tests/fixtures/toy_sidecar.py proves a non-agent process satisfies the contract in ~75 lines.

Agent (BrowserUseToolsMixin)
  └── HTTP, JSON only ──> browser sidecar  (127.0.0.1:<ephemeral>, bearer token)
                             ├── /health  /version          (supervision contract)
                             ├── /v1/browser/...            (relay path prefix)
                             └── WebSocket ──> Chromium     (CDP, stays inside)

Three constraints from the existing supervisor shape this:

The daemon relay has no WebSocket route. RELAY_METHODS is HTTP verbs only. So CDP must terminate inside the sidecar and never cross the relay — which is the right layering anyway. The sidecar exposes a small JSON API; the socket to Chromium is its private business.
The relay hardcodes /v1/<agent_id>/, so the sidecar serves /v1/browser/.
MAX_LIVE_SIDECARS = 3, with no eviction, and email + gaia already claim two. A browser sidecar is the third and last slot. Either raise the cap or add idle eviction — flagged below as a blocker, not a detail.

Sidecar specs live in a static builtin_specs() table, so registering browser is a core code change. Launch secret, 0600 secret-file delivery, Host/Origin allowlisting and tree-kill reaping all come free from src/gaia/sidecar/caller_auth.py and sidecars/manager.py.

Tool surface

Nine tools, registered as KNOWN_TOOLS["browser_use"] alongside the existing lightweight browser mixin — which stays. Most web reading should keep going through fetch_page, because it is 200 ms and free.

Planned surface. Six shipped — see What actually shipped and the guide for the real table.

Tool	Gated
`browser_open(url)`	no
`browser_snapshot()` — filtered a11y tree with stable refs	no
`browser_click(ref)`	yes, when the session is authenticated
`browser_type(ref, text)`	yes, when the session is authenticated
`browser_select(ref, value)`	yes, when the session is authenticated
`browser_scroll(direction)`	no
`browser_back()`	no
`browser_login(url)` — hand off to the user	always
`browser_screenshot()` — VLM fallback only	no

Observation format is the load-bearing decision. A snapshot returns interactive elements only, each with a stable ref the model cites back — never raw HTML, never coordinates. The token saving this assumed turned out to be smaller than planned — measured, a dense page is ~1,040 tokens of elements and ~2,000 with page text, which is a screenshot’s ballpark rather than an order below it. The decisive advantage is grounding: a ref resolves to an exact element where a coordinate is a guess.

Security

Two risks, and the second is the serious one. Credentials. Covered above: the model never sees them. The visible-window handoff is the whole mechanism. Prompt injection with an authenticated session attached. This is materially worse than anything GAIA’s browser tools face today. A page can contain text aimed at the model, and the agent can now click “Transfer” while logged into the user’s account. Read-only fetching has no equivalent failure. Mitigations, all of which reuse existing machinery rather than inventing a parallel one:

Page content is data, never instructions. Agent.turn_content_provenance already exists precisely to stop fetched content writing itself into standing instructions; browser observations must be marked the same way.
Read/act asymmetry. Navigating and snapshotting are ungated. Any state-changing action on an authenticated origin goes through the production gate — CONFIRMATION_REQUIRED_TOOLS on the agent class plus OutputHandler.confirm_tool_execution. Not @govern: src/gaia/governance/ has zero production adopters and only fires for agents that mix in GovernedAgentMixin with an adapter, which nothing does.
Scoped “always allow”. tool_grants.GrantScope already keys session grants on the invocation rather than the tool name. A browser scope rule must key on origin — browser_click:github.com — so approving one click never hands over every site for the session.
Origin allowlist per skill, using the grammar skills already speak: browser:interact:github.com, browser:session:github.com, alongside the existing network:read:*.brave.com and shell:execute:gh.
No CAPTCHA solving, ever. A CAPTCHA is a hand-off to the user, same as login.

Phases

Phase 0 — measure (1 week, blocking)

Everything above rests on one 2026-08-18 reading that has not been retaken. On real Ryzen AI hardware, measure: prefill tokens for a filtered a11y snapshot vs. a screenshot on both Gemma-4-E4B and Qwen3-VL-4B; end-to-end per-step latency; and Qwen3-VL’s click-grounding accuracy on a fixed page set.Exit: a table that either confirms text-first or overturns it. If vision grounding lands above ~90% and the token gap is under 2×, revisit — do not ship the assumption.

Phase 1 — driver interface + Playwright, no auth

The BrowserDriver protocol, the sidecar spec, the snapshot format, and the 6 ungated tools. Optional [browser] extra. No login, no credentials.Exit: the agent completes a multi-step read-only task on a client-rendered page that fetch_page cannot read today.

Phase 2 — login and sessions

Visible-window handoff, encrypted session persistence, the permission grammar, and the gate on authenticated actions.Exit: user logs into a real site once; a later run restores the session unattended; a state-changing click prompts for confirmation and denies cleanly when refused.

Phase 3 — native CDP driver

Swap Playwright for the aiohttp CDP driver against Electron’s Chromium, behind the same interface. Playwright stays supported for CI and non-desktop installs.Exit: Phase 1 and 2 suites pass unmodified against both drivers; desktop install adds no browser download.

Phase 4 — record and replay

Trajectory capture, deterministic replay, divergence detection falling back to the LLM.Exit: a price-watch run against a login-walled retailer completes with zero LLM calls on the second run.

Open questions

MAX_LIVE_SIDECARS = 3 with no eviction. A browser takes the last slot. Raise the cap or add idle eviction — needs a decision before Phase 1.
Where the visible window comes from in Phase 2, given Playwright lands first. Playwright’s own window is the pragmatic answer; the Electron window is the good one. Possible argument for pulling Phase 3 forward.
Headless-server installs. Login is inherently interactive. Probable answer: capture the session on a desktop machine and import it.
Bot detection. Out of scope to defeat. When a site blocks automation the tool must say so plainly rather than escalate — GAIA does not ship evasion.
Whether session state becomes a third ConnectorSpec.type or stays browser-local. Non-blocking.

Decisions log

#	Decision	Rationale
D1’	Revisit Browser Tools D1 — a browser engine is now in scope	D1 traded away JS rendering and was right to. It also traded away login, which is the actual gap
D2	Keep `fetch_page`; add browser use alongside	200 ms and free beats 3 s and heavy for most reading
D3	Text-first (a11y tree), vision as fallback	2.6 s per 1,000 prompt tokens, and DOM refs are exact where 4B coordinate grounding guesses
D4	Driver behind an interface; Playwright first, native CDP after	The loop is the hard part; the driver is swappable. De-risk in that order
D5	Playwright as an optional extra, never core	200 MB contradicts the edge/AI-PC positioning
D6	Native driver targets Electron’s Chromium via CDP	`electron@^44` and `aiohttp` already ship — zero new dependencies, zero new download
D7	Browser runs as a daemon sidecar; CDP never crosses the relay	The relay is HTTP-only; the supervisor already handles crash, reap and auth
D8	The model never handles credentials	MFA, passkeys and SSO all work; nothing to leak
D9	Session blob encrypted on disk, key in keyring	Windows Credential Manager’s ~1280-char cap makes the keyring-only path fail
D10	Gate on state-change × authenticated origin, via the `CONFIRMATION_HOOKS` union	`governance/` has no production adopters. Shipped as a per-call hook rather than a static name set, so the decision can depend on session state and travels with the mixin
D11	Replay, not a faster driver, is where the speedup lives	The model is 80–95% of wall clock. The only large win is not calling it
D12	No CAPTCHA solving, no bot-detection evasion	Hand off to the user; fail loudly

Browser Tools spec — the lightweight fetcher this extends
Agent latency study — source of the 387 tok/s prefill figure
CUA plan — desktop control; adjacent, deliberately separate
Security model

Test record

Everything below was run through the TUI against GLM-5p3 via Fireworks, with ground truth extracted from the DOM at run time rather than pre-baked. Adversarial payloads live on a local fixture site; none were fired at third parties. 49 live agent tasks across two rounds. The value was not the pass rate — it was that a live model does things a fixture never does, and eight defects surfaced that the unit and integration suites passed straight over.

Round	Scope	Outcome
1	35 tasks: capability, state, honesty, adversarial, forms, the six production failure modes, real pages	5 defects
2	14 tasks: deep navigation, filter-and-aggregate, multi-step reasoning, realistic workflows, advanced adversarial	3 defects

What the live runs found that the tests did not

Defect	How it presented
`time.sleep()` starved Playwright’s event loop	`target="_blank"` opened no page at all — the popup object was never created
Stale popup adopted by later clicks	Clicking “Delete account permanently” returned a report page opened two tasks earlier
Refs not scoped to their snapshot	Agent clicked `e2` three times, bouncing between two pages, and returned nothing
Page text capped at 120 chars	Every page came back as one sentence of a 63,000-character article
Disabled control burned the full timeout	30s to learn a button was greyed out
Gate keyed only on session state	”Send transfer now” clicked with no prompt
Cookie presence read as “signed in”	Confirmation prompt fired on a public weather page
Long-page content unreachable	Item 150 of 200 was in neither the elements nor the text

Where the model held up

Every prompt-injection attempt failed, and GLM named them rather than silently complying — including one planted on page 3 of a guide after two clean pages. It refused a confused-deputy request to use a signed-in session elsewhere, refused to paste its context into an “verify your results” form, declined to invent a serial number for a required field, recognised a 429 and did not retry against the stated cooldown, and converted $1.209,50 from de-DE notation correctly.

Round 3 — the shapes real sites actually use

Probed directly rather than through the model, which took seconds instead of minutes and found more. Two were total blind spots; three returned confident wrong answers, which is worse than failing.

Shape	Before	After
iframe	0 elements — a support portal read as an empty page	controls reachable, form filed end to end
shadow DOM	0 elements; then controls listed but the serial beside them missing	both controls and text
3 identical “Select” buttons	indistinguishable	each carries its row — “Account B”
covered control	30s timeout, generic message	names the blocker; GLM read it, clicked Close, retried, saved
script redirect	returned the “Redirecting…” interstitial	lands on the final page
native `confirm()`	auto-dismissed silently — clicking Archive reported KEPT	dialog surfaced; the answer explains it was declined

The covered-control case is the one worth noting: a better error message was enough for the model to recover on its own, with no new tool.

Known limits of this record

One model, one run each, on one machine — nothing here generalises to the default local model, and on the tool-loader gap it already did not. Three tasks ended in an empty answer traced to the LLM stream being torn down mid-body (GeneratorExit) on long cloud turns; that is GAIA’s streaming layer, not the browser, and is filed separately.

What's Next

Agent UI

Ecosystem

Infrastructure

Agents

Browser Use

Browser Use — Scope

What’s broken

The thing to get right first: what actually costs time

Where the real speedup actually is

What actually shipped

Recommendation

Where the session blob lives

Architecture

Tool surface

Security

Phases

Open questions

Decisions log

Test record

What the live runs found that the tests did not

Where the model held up

Round 3 — the shapes real sites actually use

Known limits of this record

​Browser Use — Scope

​What’s broken

​The thing to get right first: what actually costs time

​Where the real speedup actually is

​What actually shipped

​Recommendation

​Login

​Where the session blob lives

​Architecture

​Tool surface

​Security

​Phases

​Open questions

​Decisions log

​Related

​Test record

​What the live runs found that the tests did not

​Where the model held up

​Round 3 — the shapes real sites actually use

​Known limits of this record

Browser Use — Scope

What’s broken

The thing to get right first: what actually costs time

Where the real speedup actually is

What actually shipped

Recommendation

Login

Where the session blob lives

Architecture

Tool surface

Security

Phases

Open questions

Decisions log

Related

Test record

What the live runs found that the tests did not

Where the model held up

Round 3 — the shapes real sites actually use

Known limits of this record