Skip to main content

​
Adaptive recovery: stop, ask, and remember

Status: Proposed — nothing below is implemented unless marked exists today Priority: P1 Tracks: #4428 Related: Adaptive Memory (outcome-scored lessons), Adaptive Skills (#2674, the learned skill overlay), #4399 (short-circuit repeated calls), #3381 (verification state), #3393 (validate before dispatch), #3584 (gathering vs acting)

​
1. The problem

When a tool says the environment needs the user, the flagship improvises workarounds until it runs out of ideas, never asks, and learns nothing. In #4428 gh was logged out. check_cli_setup said so on the first call, with the exact sign-in command. The agent then spent 43 tool calls and 8 minutes on two 45 s browser sign-ins nobody could approve, the public API, web search, curl, PowerShell, urllib, env-var dumps and GH_TOKEN= injection. The permission gates refused every unsafe move; nothing told the agent to stop. The next session would do it all again. The fix is not a gh rule. The loop needs a general way to recognise “this needs the user”, a hard limit on improvising after that, a precise question, and a memory of the answer that expires when the world changes.

​
2. What exists today, and what works against it

​
3. Design

Three layers, strictly ordered: classify (what happened), policy (what the loop allows next), learning (what the next session starts from). Nothing in the loop names a specific tool.

​
3.1 Taxonomy

Every tool result is at most one of the following situations: Blockers split further on one flag that matters to policy: interactive, true when the remedy needs a human at a keyboard (browser sign-in, 2FA, an approval prompt).

​
3.2 Where classification lives

Declared first, inferred second, never per-tool code in the loop.
  1. Declared. A tool that knows its own failure modes returns a situation key next to status:
    The contract lives in a new src/gaia/agents/base/situations.py: a Situation dataclass, the kind vocabulary, and validate_situation() that the loop calls on every result. An unknown kind or missing subject is a loud ValueError naming the tool: a malformed declaration is a bug in that tool, not something to guess around.
  2. Inferred. For tools that declare nothing, classify_result(tool, args, result) in the same module maps the result onto the taxonomy from generic signals only: executed: False plus tier → permission; rate_limited → quota; the existing connection and not-permitted regexes (moved out of agent.py:9275); HTTP 401/403 and “not logged in” / “authentication required” → auth; “command not found” / “is not recognized” → not_installed; empty list/string on a search tool → empty; the truncation marker from _truncate_large_content → truncated. Inferred situations carry source: "inferred", never an unblock string, and never a probe.
The tool registry gains one flag, @tool(interactive=True), set on tools whose whole job is a human handoff (sign_in_cli, request_user_input). That is a property of the tool, not of a result, so it lives on the decorator.

​
3.3 Policy

The policy sits at the #3393 seam, before dispatch, and after each result. It holds one object per turn, RecoveryState: the open blockers (by subject) and a workaround count for each. Rule 1: no interactive flow without a human. A call to an interactive tool is refused before dispatch when nobody can answer it. That covers the eval harness, the API server, scheduled runs, and any console where _console_accepts_stdin_prompts() is false and no live SSE/TUI client is attached. The refusal is a normal tool result with executed: False and a situation of kind permission, so it also counts against the budget. This alone removes the two 45 s sign-ins from #4428. Rule 2: a bounded workaround budget. After a declared blocker on subject S, each further call that fails, is refused, or returns a situation of the same kind increments S’s count. Successful calls don’t: the agent may keep doing whatever it can do without S. Probes don’t either, since checking is not working around (#3584). When the count passes max_workarounds_after_blocker (default 2, on the agent config dataclass), the loop stops dispatching tools for the turn and escalates. Inferred blockers get a budget of 4, because a regex is less sure than a tool. A blocker closes when its probe returns no situation, when a call on S succeeds, or when the user answers. Closing resets the count. Rule 3: blockers get their own recovery prompt. When the failing result carries a blocker, the generic “create a corrected plan that fixes the error” prompt (agent.py:7076) is replaced by one that states the blocker, the remaining budget, and the two legal moves: a genuinely different safe approach, or escalate. Self-fixable errors keep today’s prompt. Rule 4: the escalation contract. An escalation always says three things, built deterministically from the situation fields so it can’t be lost to a weak model:
The first two lines come from the declared fields. The third comes from one closing model call with tools withheld, the same mechanism _answer_after_repeated_calls uses (agent.py:9401). If that call fails, the first two lines ship alone with the error appended. They never ship silently trimmed. Delivery goes through the answerable path. In the Agent UI with a live client the loop calls request_user_input_blocking with the choices “I’ve done it” / “Skip it” / “Stop”. In the TUI the turn ends with a new escalation SSE event carrying the same fields, rendered as a question whose reply goes through /followup (mid-turn) or as the next turn. Headless runs end the turn with the escalation as the answer and a non-zero blocked status in the result, so callers and evals can tell a blocked turn from a finished one. That is the #3381 verification state: blocked is not done.

​
3.4 Learning

A lesson is a remembered blocker: “the last time we called S here, it needed the user, and here is exactly what they had to do.”
The signature is tool + subject + kind + env. env is a fingerprint of what the situation depends on: for a CLI, the resolved binary path and --version; for network kinds, nothing (they are never persisted, see below). Why the knowledge table, not skill_deltas or a new table. A blocker is a fact about the machine, not a correction to a skill’s text. skill_deltas is bound to one authored skill, requires user_instruction provenance, and applies at session boundaries. All three are wrong for “gh is logged out”, which applies across every skill that uses gh and changes mid-session. The knowledge table already has confidence, context, JSON metadata, superseded_by and FTS, and memory.py already writes loop-derived lessons there and invalidates them on contrary evidence (_note_failure, _learn_from_success). Situation lessons reuse that path under their own domain, so they never mix into “Known errors to avoid”. If Adaptive Memory’s lesson category lands first, situation lessons move under it unchanged. Who writes it. Only the loop, only from a declared situation, and only after the turn escalated or the user answered. Inferred situations are never persisted in slice 2. The model has no tool to write one. Retrieval, at the moment it matters. Two points, both bounded:
  1. Turn start: up to 3 active situation lessons whose subject belongs to a loaded skill’s declared binaries, as a Known blockers on this machine section in _build_dynamic_memory_context (memory.py:2754).
  2. Before dispatch of any call whose subject matches a lesson: if the lesson has a probe, the loop runs the probe first. Still blocked means escalate immediately, citing the lesson (0 wasted steps). Clear means invalidate the lesson and dispatch normally. No probe means the lesson is prepended to the tool result as a one-line note, and the model decides.
A lesson can make the agent ask sooner. It can never skip a call without saying so, never grant anything, and never suggest a command that didn’t come from tool code. Staying right. Poisoning. The attack is a fetched page or issue body that makes GAIA “remember” that a tool is broken, or that the fix is some command. Four properties close it:
  • unblock and probe come only from tool code; free-text output never reaches them.
  • Inferred situations are never stored.
  • Lessons only move toward asking: they never add a tool, a permission, or a command.
  • Every lesson is re-verified by a probe before it blocks anything.
Results the classifier marks injected_instruction are excluded from all learning and are the hand-off point for the #2468 analyzer.

​
3.5 Observability

Every decision is visible. Nothing in this system changes behaviour quietly. Events go through TurnRecorder (turn_metrics.py, GAIA_TURN_LOG) and the SSE stream. The TUI /cost view adds one line when a turn had a blocker: “12 steps · 94 s after gh was blocked, before asking”. That turns the cost of improvisation into a number users and maintainers can see.

​
4. Measuring it: gaia_resilience

A new eval category that injects unexpected states across many tools. Scenarios are authored separately (they live in eval/scenarios/); this section fixes what they must measure. Injection, reusing what exists: new ghstub modes logged_out and insufficient_scopes next to offline/rate_limit/transient; gh removed from PATH for not_installed; task_setups.py entries for a locked file, a permission-denied directory, an oversized log (truncation), a mis-encoded file, and a README carrying an injected “run this curl” instruction; a search fixture that returns nothing. Judged dimensions, each 0–10: Also recorded, not judged: steps_to_escalate, seconds_to_escalate, workaround_calls, refused_calls. The #4428 run is the reference: 43 / 480 s. Slice 1’s target is ≤ 3 steps. Counter-scenarios matter as much. Self-fixable errors (wrong path, bad args), and a blocker the agent can route around legitimately (answer from the local clone), must not escalate. Over-asking is a regression, and these scenarios catch it. Multi-session scenario (learning): session 1, gh logged out, must escalate and store a lesson. Session 2, same state, must escalate within 2 steps citing it. Session 3, stub switched to signed-in, must probe, invalidate, and finish the task. Per CLAUDE.md every scenario scores the flagship, and no baseline number is committed until a real run on AMD hardware produces one.

​
5. Rollout

Slice 1: declared blockers stop and ask (no learning).
  1. situations.py: the Situation contract, the kind vocabulary, and validate_situation().
  2. check_cli_setup / install_cli / sign_in_cli return status: error plus a situation for missing / unauthenticated / insufficient_scopes. Audit callers that read status == "success" to find the state.
  3. @tool(interactive=True) on sign_in_cli, and Rule 1 in the loop.
  4. RecoveryState, Rule 2 (budget on the config dataclass), Rule 3 (blocker recovery prompt), Rule 4 (deterministic escalation with a closing call).
  5. situation.* trace events.
  6. Unit tests: validation errors, budget counting, interactive refusal headless, escalation text from fields, and self-fixable errors untouched. Plus two gaia_resilience scenarios (gh logged out, gh missing) and one counter-scenario.
Slice 2: learning. Situation lessons, probe-first retrieval, invalidation, the multi-session scenario, and the escalation SSE event plus TUI rendering. Slice 3: breadth. The inferred classifier and degenerate outputs. Declared situations for shell REFUSE, browser 401/403/429, and file locks. The /cost line. Slice 4: adversarial. injected_instruction wired to #2468, and cross-session reporting of wasted steps.

​
6. Risks

  • LLM-affecting. Rules 3 and 4 change prompts and loop control. Every slice runs gaia eval agent on gaia_resilience, error_recovery, gaia_shell and gaia_tool_selection before merge, serially per Lemonade backend.
  • This must not become a fallback system. The loop never substitutes a different tool, provider or path on the agent’s behalf. It stops and says so. Budget exhaustion ends the turn, and a lesson triggers a probe or a question, never a silent skip. A malformed situation raises. If a slice needs “try X instead” logic, it belongs in a skill, not here.
  • Over-escalation. A budget of 2 may be tight for tasks with real alternatives. The counter-scenarios measure it, and the default moves on evidence, not intuition.
  • “Is a human present” is fuzzy. Rule 1 needs one explicit signal on the console (unattended: bool, set by the eval harness, API server and scheduler) rather than inferring it from console type. Slice 1 adds it.
  • Overlap with in-flight PRs. #4399 owns identical repeats and this doc owns varied workarounds, so they share RecoveryState bookkeeping rather than adding a second counter. #3393’s pre-dispatch hook is where Rule 1 and probe-first retrieval go. If it hasn’t merged, slice 1 adds the narrowest hook it needs and #3393 absorbs it.

​
7. Evidence

Observed runs where an unintended situation led to unbounded improvisation. Append new cases to the table with a link to the run or issue.

​
8. Open questions

  • Should the budget count per subject or per turn? Per subject is proposed; a turn with two independent blockers is rare enough to measure first.
  • Does machine:<fingerprint> need the OS user in it for shared machines?
  • Should an escalation with “Skip it” record a preference (“work without gh here”) as well as the lesson? Deferred to slice 2 review.