InjectionWatch
A public, human-reviewed feed of prompt injections hidden in real web pages and aimed at AI agents — measured on 426,318 Common Crawl pages, where the honest count was seven.
- Type
- Artifact
- Where
- Personal
- When
- Oct 2026 → Present
- Duration
- 1 mo
- Status
- Live data · v0.3
- Source
- Private · available on request
The problem
AI agents read what people never look at: HTML comments, meta tags, hidden elements, attributes, response headers. Text placed there can address the agent directly, and the person using the agent never sees it. Google and one large academic crawl had both measured this on the public web — as one-off snapshots.
There was no PhishTank for agents: no URL-level feed, with evidence and a takedown path, that a tool could ask before reading a page. And there was no defensible number for how often a random page actually carries one, because a detector that has never met the real web will tell you whatever its fixtures taught it.
What I built
Every page becomes candidates — a piece of text plus where it lives: comment, meta tag, CSS-hidden element, aria-hidden, attribute, JSON-LD, invisible Unicode, HTTP header. 101 rules in ten languages score them, scaled by context: a page that discusses prompt injection, a quoted mention or a code block counts for less. A candidate fires only above a threshold, and only on a standalone rule or two substantive ones.
The first live crawl never fetched a page — the sandbox had no outbound access, and the crawler's circuit breaker stopped after eight failures instead of retrying under another identity. So the measurement moved to Common Crawl: WARC files streamed record by record across nine worker processes, resumable by byte offset. Twenty random files from CC-MAIN-2026-39 came to 426,318 HTML pages, scanned in just over two hours on a laptop.
Nothing is published unreviewed. I labelled every injection-like finding by reading the saved page, and only reviewed true positives reach the feed: a static dashboard with per-host lookup shards so a URL can be checked with no server, JSON, CSV and STIX exports, an API whose URL check never fetches the URL, an MCP server, and a Manifest V3 extension that runs the same rules locally.
Screens
What I considered
Rejected. 426,318 pages produced 104.6 million candidates — a bill, a latency budget, and a third party reading every page. A model stays an optional, local second stage that can veto a finding but never invent one.
Rejected. At 0.44 precision, more than half of the feed would be false accusations against real sites. The detector is triage; a person is the classifier.
Chosen. Every finding explains which rules fired and where, the held-out pages were hashed before any tuning touched them, and the feed grows only as fast as one reviewer can read. The cost is scale and unmeasured recall — in exchange for a precision figure I can defend line by line.
Results
- Scanned 426,318 HTML pages from 20 random WARC files of CC-MAIN-2026-39 and hand-labelled all 16 injection-like findings: 7 were true injections, on 5 sites.
- Detector precision 0.44 (95% Wilson 0.23–0.67) on that fresh sample, up from 0.027 when the original v0.1 rules first met the real web.
- True injections at 1.64 per 100,000 pages (0.80–3.39), none of them exfiltration or credential phishing; AI opt-out tokens, counted as a separate signal, at 61 per 100,000.
- On 46 held-out pages frozen before any tuning: precision 0.33 and recall 0.08 for v0.1, against 1.00 and 0.83 static and 1.00 and 1.00 rendered for v0.3.
- The feed's first build holds only the 7 reviewed true positives, with JSON, CSV and STIX exports; it is built and not yet deployed.
Architecture
In hindsight
The evidence is thin, and the write-up says so first: one crawl, one reviewer, seven true positives, and recall never measured. Only pages the detector raised were read, so 1.64 per 100,000 is a lower bound on what a static rule scan can see, not on what the web contains. The expensive lesson came earlier — 0.95 precision on held-out pages I had written became 0.027 on pages I hadn't, because my negatives tested my imagination rather than the web. Next is a second annotator, seeded canaries to measure recall, and real negatives collected before the next rule is written.