# The web as ground truth

## What a model remembers

A language model has two kinds of memory.

Parametric memory lives in the weights, and is created during training: broad knowledge, patterns, associations, a rough impression of everything the model has seen.

Contextual memory is part of the model prompt: the specific documents, facts, and history the model is given as evidence to produce a specific answer.

The two systems have complementary properties.

Parametric memory is broad, fast, and useful, but lossy.

Weights are a form of compression. Training preserves patterns and loses detail, often the details that become valuable later: a specific legal precedent, a precise figure in a chart, anything too obscure to be well represented in the training run.

A frontier model can explain how a new class of cancer drugs works. Ask it for the overall survival figure from a Phase 3 readout reported last month, and it cannot answer. It also cannot reliably tell you whether a number came from the trial's primary endpoint or from a secondary analysis someone summarized on X.

Parametric memory is also stale. Retraining a frontier model costs hundreds of millions of dollars (Epoch AI estimates xAI's Grok 4 run at $490 million; Stanford's AI Index puts Llama 3.1 at roughly $170 million), and the largest runs take three months or more before the model ever ships. Weights cannot keep up with a world that changes daily. The richest training data can't help a model say what moved the price of oil yesterday, or which amendment a European regulator certified an hour ago.

Parametric memory alone falls short for work that demands recency and precision, which is most work that drives action in the real world.

Contextual memory can invert these properties. Given the right content, it can be exact, current, and traceable to its source.

But it has a challenge of its own: finding the right content, extracting the relevant parts, and presenting them in a form the model can use without overwhelming its limited context window. Retrieval is becoming the central bottleneck in AI.

## Retrieval by default: Harnessing the web for intelligence

The open web is the largest, freshest, broadest record of human knowledge in existence. Scientists, regulators, journalists, and companies all publish to the web before any other medium. Without it, a model is capped at whatever it memorized before its training cutoff.

All knowledge work depends on it. The best employees look things up: they check current prices, read recent filings, compare sources, and stay current in their field. Their raw intellect matters, but their impact depends on access to the world's information.

The same logic applies to AI. A model cut off from the web is a knowledge worker with vast intellect and no internet access. It cannot learn new things, verify a claim, or check the veracity of a source. You wouldn't impose that constraint on an employee. The same standard should apply to the AI systems used throughout your company.

The obvious objection is that the web is also the largest record of junk in existence: SEO farms, affiliate sludge, and now machine-generated text at industrial scale. Graphite estimates that AI-written articles crossed half of all newly published articles in late 2025. True, and it is the argument rather than the rebuttal. Ground truth was never the web as-is. It is what a retrieval layer built for verification extracts from the web: the filing underneath the coverage, the primary source under the aggregator, the number with a URL attached. As synthetic content grows, provenance becomes the product.

Retrieval belongs in the default architecture of any AI system that handles real-world decisions.

## Agents don't think in web pages

Retrieval augmentation for AI systems began as little more than the human web search that preceded it: ten blue links, fetched by a thin wrapper around a search engine and dropped into the model's context for it to sort out.

The web is acquiring its second user, and everything about it was built for the first. Decades of design optimized every page for human eyes: most of what a page delivers is scaffolding around how people find and view content. For a model, anything that isn't information is overhead: tokens to sift through, pay for, and discard before arriving at the kernel of knowledge it needs.

Even a trivial query like "who won the French Open yesterday" surfaces almost entirely discardable content: HTML markup, SEO filler, navigational headers, and breathless tournament coverage. Measured directly, the raw HTML of a typical article page runs about 45,000 tokens; the extracted answer runs about 400. Retrieve whole pages and the model's context budget is spent on the packaging.

A phenomenon called context rot makes this worse: model accuracy degrades as context grows, even on tasks the model handles perfectly at short lengths. GPT-4o fell from 99.3% to 69.7% on the same retrieval task once the context reached 32,000 tokens. NVIDIA's RULER benchmark found only half the models tested held up at 32,000 tokens, whatever their advertised window. And Chroma's study of 18 frontier models showed that a single well-placed distractor measurably degrades answers, with additional distractors compounding the damage. The larger the haystack, the more it holds things that resemble the needle.

Consider what the atomic unit of retrieval actually is. A biotech analyst needs the overall survival figure from last month's readout, not the press release built around it. A CRM enriching a million records needs one field per company: the employee count, the pricing change, the new security chief. A coding agent needs the four lines of the changelog that explain the breaking change. Across these workflows, the atomic unit is the snippet of pertinent evidence: the sentence, the paragraph, the table row, the chart in a PDF, the code block that nudges the model closer to a correct answer. The webpage is too coarse a unit for the job.

Because the atomic unit can be as small as a single figure traced back to its source, retrieving snippets instead of full pages frees an order of magnitude of context for evidence and reasoning. Producing those snippets at web scale is a prerequisite for a context layer that can ground agents at scale.

## The context tax: paid in cost, latency, and accuracy

A page full of SEO is a minor annoyance for a single user. At the scale agents operate, the overhead becomes a hard ceiling on what they can do. Anthropic reports that a single agent consumes about four times the tokens of a chat session, and multi-agent systems about fifteen times. A deep research run reads hundreds of sources. Cursor has run hundreds of coding agents concurrently against a single project. Clay's customers rehydrate CRMs of ten million records. Every junk token in those loops becomes a tax.

It is a cost tax, and the arithmetic is blunter than any curve: input tokens are billed on every call, and an agent re-sends its context at every step. A 45,000-token page carried through a twenty-step trajectory gets paid for twenty times, by every subagent that touches it. Prompt caching discounts the rate (cached reads run about a tenth of list price) but doesn't erase the bill, and freshly retrieved content is a cache miss by definition.

It is a latency tax, because attention, the mechanism transformers use to relate tokens to each other, scales with the square of context length. The per-token price you're billed stays flat; the compute underneath does not, which is why Google charges double the input rate once a Gemini prompt passes 200,000 tokens.

And it is an accuracy tax, because of finite context windows and context rot.

Reasoning-heavy models already spend tokens planning, checking, and synthesizing before they answer. Make them grind through entire web pages on top of that and they slow to a trickle.

Ask a frontier model to play to its strengths instead: help it find the right pages, strip them to the relevant passages, and let it operate on information-dense snippets. That density becomes an order-of-magnitude multiplier on the efficiency of every LLM call.

With a capable retrieval layer, agents spend their reasoning budget on the work itself rather than on cleaning up the input. They complete tasks faster, more accurately, and at a fraction of the cost. Cheap enough that reaching for the web becomes the default for every agent.

## Built for the median question

If grounding matters this much, why isn't the web search offered by the model labs sufficient?

For plenty of use cases, it is. But it's tuned for the median question: speed, common queries, and surface-level insight, the kind that dominates everyday chat. Labs hill-climb on user feedback, and user feedback rewards a fast answer over a thorough one.

And nobody is optimizing your token bill but you. Lab search is priced for the lab's own surface: about $10 per thousand searches, plus the retrieved content billed as input tokens. On some OpenAI models, every search call meters a fixed 8,000-token block whether or not the page helped. A frontier model can extract an answer from a raw page, and for a consumer chat product that's fine. For a fleet of agents running millions of retrievals a month, it is a staggeringly expensive use of frontier intelligence, and the bill lands on you. Enterprises have noticed: in a 2025 IDC survey, 96% of organizations deploying generative AI said costs ran higher than expected, and 71% said they had little control over where those costs came from.

A one-size-fits-most product can also offer only so much control. Standardization trades away the ability to preserve provenance from index to snippet, or to decide which parts of the web to trust more or less.

Serious enterprise and specialized agentic workloads demand more: multi-step research where missing a single source can invert the conclusion of a drug trial, and agents that inspect hundreds of pages per task without drowning the model in tokens.

## The past, present, and future of retrieval

With each generation, models and grounding grow more interconnected. ChatGPT launched with no web access and training data 14 months stale. The research systems came first (OpenAI's WebGPT was following links and citing sources in 2021); the products followed: Bing wired chat into search in early 2023, ChatGPT Search arrived in late 2024, and labs now train models end-to-end on when and how to search. Retrieval itself keeps evolving, from dense snippets toward agentic task execution: subagents specialized in long-horizon retrieval that run dozens of lookups to finish a single task. When Anthropic built its multi-agent research system, token usage alone explained 80% of the variance in performance across its evals. What an agent can effectively consume determines what it can do.

The step after agentic retrieval is still on the research frontier, and no architecture has won yet. But the retrieval layer's properties are already clear: its latency, density, fidelity, and provenance. Those four properties will bound every AI system built on top of them.