Dynamic Tool Loader

​

Why this exists

1. Cold-start latency (the user-felt problem). On local inference the model must prefill the entire system prompt before emitting the first token. The first query of a session is the worst case — nothing is cached, so the whole prompt is processed from scratch. A large tool block directly inflates first-turn time-to-first-token (TTFT). Trimming it is the most visible win: users perceive a slow first reply, never a token count.
2. Scaling headroom (the architectural problem). As agents gain tools (MCP connectors, skills, new mixins), the prompt grows linearly with the registry. Today’s absolute waste may be modest, but the slope is the problem: every tool added taxes every turn forever. This mechanism breaks that link — prompt cost scales with tools loaded, not tools registered.

​

Decided design

┌──────────────────────────────────────────────────────────┐
│ CORE  — always loaded. The functional floor.             │
│   e.g. memory, read/write, search, finish (set TBD).     │
├──────────────────────────────────────────────────────────┤
│ SEMANTIC  — primary mechanism.                           │
│   embed(conversation) → cosine vs. embedded tool docs    │
│   score tool docs ≥ τ → open matched tools' bundles.     │
├──────────────────────────────────────────────────────────┤
│ ESCAPE HATCH  — safety net for semantic misses.          │
│   recovery is PATH-DEPENDENT (see "Escape hatch"):       │
│   non-tool-calling → unlisted tool still executes;       │
│   native tool-calling → explicit load_tools meta-tool.   │
├──────────────────────────────────────────────────────────┤
│ SKILL  — future signal, gated on #887 (Part 3).          │
│   recall_skill(goal) → union matched skills' tools.      │
│   precision booster; returns [] cleanly until #887.      │
└──────────────────────────────────────────────────────────┘
   loaded = CORE ∪ SKILL ∪ SEMANTIC ∪ (escape-hatch additions)

​

Bundles are cohesion groups, not triggers

​

The escape hatch — free on one path, explicit on the other

• Non-tool-calling models — recovery is nearly free. The model emits free-form tool names, and Agent._execute_tool validates against self._tools_registry — the full registry — with unknown-name resolution (MCP-prefix stripping, bare-prefix candidate lists) already built in. So if the loader filters only the prompt and leaves the registry whole, a tool the model names that semantic didn’t surface still executes. The escape hatch’s only extra job here is re-surfacing that tool’s bundle in later turns.
• Native tool-calling models — recovery must be explicit. These models are physically constrained to the schemas passed in tools=; they cannot emit a call for a tool that isn’t listed. Reactive-via-execution is impossible. This path needs an always-loaded load_tools(bundle) meta-tool (in the schema list every turn) plus a short menu of bundle names + one-liners; the model calls it, and the next turn’s tools= includes the expanded set. On a small local model this is the less-reliable path — measure whether it actually fires.

​

When memory is off, all tools load (the legacy path)

​

Correcting the original spec

​

The two render paths (most important correction)

• Non-tool-calling models → _format_tools_for_prompt() renders text into the system prompt (cheap, one line per tool).
• Tool-calling models → _build_openai_tool_schemas() (via the _openai_tools property) passes full JSON function schemas as the tools= API param. This is where the real tokens live.

​

KPIs

​

The cache-thrash trap (TTFT’s flip side)

• After turn 1 the system-prompt prefix is KV-cached — that is why turns 2+ are fast. If the loader recomputes selection every turn and the tool set changes, it invalidates the cache and re-prefills every turn, turning a first-turn win into an every-turn tax.

1. Select once at conversation start (or change rarely), not aggressively per-turn. Stable selection preserves both the cold-start win and warm-cache reuse.
2. Place the dynamic tool block at the end of the prompt, after all stable instructions. KV cache is prefix-based: everything stays cached up to the first byte that differs, so the volatile part must come last. Note this is a text-path lever — native tool-calling models carry tools in the tools= param, not the prompt body, so there the only lever is set stability (rule 1), not ordering.

​

Phased build

​

Part 0 — Measure first (do this before any loader code)

• A reproducible harness reports tool-prompt token cost for ChatAgent on both the text path (_format_tools_for_prompt) and the native path (_build_openai_tool_schemas).
• Prompt-cost slope (tokens per added tool) is quantified for both paths.
• Baseline first-turn TTFT is recorded for the unfiltered agent.
• A go/no-go note states whether the measured cost justifies the loader, and sets the concrete TTFT-reduction target Part 1 must hit.

​

Part 1 — Selection + dual-path filtering

• Wire selection into _compose_system_prompt; filter both _format_tools_for_prompt and _build_openai_tool_schemas.
• CORE always-on + semantic via _embed_text; bundles as cohesion groups.
• Fix the caching seam: recompute when selection changes; tool block last.
• Embedder down → session-disable fallback (mirror memory v2), logged.
• Memory disabled or loader toggle off → expose all tools (legacy path). See When memory is off.
• Leave _execute_tool on the full registry. This alone gives the free non-tool-calling recovery (an unlisted tool the model names still runs) — so Part 1 is not without a safety net, even though the explicit escape-hatch machinery lands in Part 2.
• Default-off toggle; scope to ChatAgent doc.

• First-turn TTFT drops by at least the Part-0 target with the loader on.
• Tool recall ~100% — every tool the unfiltered baseline called was available when the loader-on run called it (the merge gate).
• Task success within ~2% of the unfiltered baseline on gaia eval agent.
• No later-turn TTFT regression — turns 2+ stay warm-cached (selection stable).
• Both render paths verified — text block and native tools= schemas measurably shrink, not just one.
• All three off-states revert to all tools — memory disabled, loader toggle off, and embedder unavailable each expose the full registry (the last logs).

​

Part 2 — Explicit escape hatch + tuning

• Add bundle re-surfacing + a discoverability menu of bundle names, and the load_tools meta-tool that native tool-calling models need (the free recovery from Part 1 covers only the non-tool-calling path).
• Instrument escape-hatch activation rate as the semantic-threshold tuning dial.

• A native tool-calling model recovers a semantically-missed tool via load_tools within one extra turn (demonstrated end-to-end).
• The non-tool-calling free-recovery path is verified (an unlisted tool the model names still executes).
• Hard recall failures = 0 — with recovery in place, no task fails because a needed tool was permanently unreachable.
• Escape-hatch activation rate is logged per session and usable as the threshold-tuning signal (rising rate ⇒ τ too strict).

​

Part 3 — Skill-driven signal (gated on #887)

• When recall_skill(goal) matches the user’s goal, union in the matched skills’ tools_required ahead of semantic results.
• This is high precision, low recall: it fires only for goals solved before, but when it does it is exact — the tools come from a recorded successful run, not a similarity guess. It complements semantic match (high recall, lower precision), it does not replace it.
• Same retrieval mechanism, different corpus. Skills are stored semantically in the same MemoryStore as #606 memory — each procedure carries an embedding, and recall_skill(goal) is a vector search over procedures.embedding. So this tier reuses the same embedder as the semantic tier; it just searches learned procedures instead of tool docs.
• Additive, not a reshape. It slots in as one more input to the same selector: loaded = CORE ∪ SKILL ∪ SEMANTIC ∪ escape-hatch. Precedence is CORE > SKILL > SEMANTIC. Nothing in Parts 0–2 changes.
• Soft dependency. Until #887 ships, this signal returns [] cleanly and the loader runs on CORE + semantic exactly as in Parts 1–2 — so Parts 1–2 must not assume it exists.

• When a skill matches the goal, its tools_required are loaded ahead of semantic results (verified with a fixture skill).
• Measurable precision lift on tasks with a known matching skill — fewer escape-hatch activations and/or higher recall than the Part 1–2 baseline.
• Graceful absence — with #887 disabled/absent, the signal returns [] and Parts 0–2 behavior is byte-for-byte unchanged (regression test).
• No regression when no skill matches the goal (falls through to semantic).

​

Open questions for the implementer

1. Bundle definitions and CORE membership — which tools group together for ChatAgent, which tools are always-on CORE (the diagram list is illustrative, not decided), and where the bundle→tools map lives (no such grouping exists today; KNOWN_TOOLS is mixins, not bundles).
2. Similarity threshold τ and how many tools/bundles to load before a cap.
3. Selection stability policy — select once per session, or re-select on a detected topic shift? What triggers a re-selection without thrashing the cache?
4. Query construction — what text gets embedded as the match query: the current user turn only, the last N chars of the conversation, or a blend? This directly drives recall.
5. Config / toggle location — where the default-off switch and any tunables live (e.g. ~/.gaia/config.toml [agent.tool_loader], env var, or constructor arg).
6. Part-0 methodology — exact harness for measuring TTFT and slope (extend gaia eval agent, or a standalone bench?).

​

Current state of the code

​

Dependencies

• #606 (memory v2) — landed. Provides MemoryMixin._embed_text (nomic-embed-text-v2-moe-GGUF, 768-dim) already mixed into ChatAgent. Hard dependency for semantic selection; no new infra needed. Note memory is user-controllable — when the user turns it off, the loader reverts to the legacy all-tools path (see When memory is off).
• #887 (procedural memory / skills) — not landed. Provides recall_skill and SKILL.md tools_required frontmatter. The skill tier is deferred to Part 3, not in v1 (Parts 0–2); it slots in additively once #887 lands and returns [] cleanly until then.