Speculate
Speculative tool execution for AI agents — a lossless TypeScript library, and a measured negative result: on 8,754 real trajectories it saves 0.1–0.2% of wall time, and a perfect predictor could save about 1–2%.
- Type
- Artifact
- Where
- Personal
- When
- Oct 2026 → Present
- Duration
- 1 mo
- Status
- Research · 1.0.0-rc
- Source
- Private · available on request
The problem
An agent turn alternates two phases: the model generates, then a tool runs, and each sits idle while the other works. CPUs solved that shape of problem decades ago with speculative execution, and two recent papers reported real latency gains from doing the same for agents.
What was missing was something a developer could put in front of their own tools with a straight answer to the question that matters — what happens when it guesses rm -rf? And underneath that, whether the speed-up survives contact with real agents at all.
What I built
While the model thinks, predictors guess the next call — an n-gram over tool sequences that answers in microseconds, and an optional LLM predictor whenever it answers. A guess runs early only if the tool is read-only: unknown means side-effecting, and a mutating word in a tool's description vetoes its read-only hint. A result is served only on an exact match of tool name and canonical arguments, so speculation can make a call faster but never different.
Every write bumps an epoch and discards speculative state before and after it runs, and validators fingerprint external state before a speculative read and check it again at use. Network tools stay off unless opted in, because a prompt injection in a tool result could steer the predictor into an exfiltration call the agent would never make. Adapters cover the Anthropic tool loop, the Claude Agent SDK, and any MCP server as a no-code stdio proxy.
A deterministic simulator said up to about 11%. So I replayed 8,754 public coding-agent trajectories — 330,942 tool calls from five Hugging Face datasets — split by trajectory, never by step. The n-gram hit 5.0% top-1, and the hit rate turned out to be the smaller problem: models think for a median 5.3 s against a 0.45 s median tool, only about 41% of calls are safe to pre-run, and those are the fastest ones.
Screens
What I considered
Rejected. grep foo ./src and grep foo src look alike and can return different output. Exact matching on a SHA-256 of canonical arguments is what makes the library lossless rather than usually right.
Rejected. Its agent thought for less time, its tools were slower, and its traces kept the tool results the predictor leans on — every assumption favoured speculation. It is the right instrument for correctness and the wrong one for benefit.
Chosen. The library stays useful where tools are slow and read-only, and eval:traces gives anyone the perfect-predictor ceiling for their own workload before they build anything. The cost is a headline of 0.2% instead of 11%.
Results
- Replayed 8,754 public coding-agent trajectories (330,942 tool calls): 5.0% top-1 and 6.4% top-3 exact hits on held-out trajectories, an estimated 0.1–0.2% of wall time saved, and a perfect-predictor ceiling of 0.8–1.8%.
- Traced the bound to Amdahl's law: a 5.3 s median think time against a 0.45 s median tool call, with only about 41% of calls safe to pre-run.
- Caught a metric that lied: a no-op think tool raised top-1 from 5.0% to 8.2% while saving no time, so it is excluded from the headline.
- Lossless under test: byte-identical transcripts with and without speculation on 200 of 200 simulated tasks per variant, and 2,000 hostile predictions that never ran a non-read-only tool.
- 136 Vitest tests at 96% line coverage, a dual ESM/CJS build, and an MCP proxy that serves repeat 400 ms reads in about 3 ms over real stdio.
Architecture
In hindsight
The real evaluation covers coding agents only, from public datasets that strip tool results — which disables the predictor's best trick — and there is still no live A/B; the LLM predictor has never been measured on real traces at all. The lesson was ordering. The most informative number in the project was the perfect-predictor ceiling, and it needs only timings and safety classes, no predictor. Computed first, it would have pointed me at slow read-only workloads — search-and-fetch loops, remote sandboxes — before I built the predictor it ended up bounding.