Agents running on mcp-data-platform do something the design never asked for: they store world state as knowledge. “Three monitors provisioned.” “Zero monitors exist yet.” These are observations, and the capture tool persists them as durable notes. State changes and notes do not, so I had a specific worry: stale observations would accumulate in the knowledge layer, and agents in later sessions would inherit them as facts instead of challenging them against the live system. I was concerned enough that I drafted schema fields for volatility classes, observation age, and recheck cost before measuring anything. Then I measured, and the premise fell apart at first contact with data. This post covers the second report in the mcp-data-platform benchmark series: the hypothesis it pre-registered, the data that falsified it, and the two-factor structure that replaced it.

The headline is a pair of inversions. On the model tier I actually deploy (Sonnet 5), delivered knowledge about checkable world state never decides an answer: the agent re-derives the state from the live API in 32 of 32 attempts, at every price point I could construct, with a median effort delta of exactly zero calls versus agents handed nothing at all. The staleness pollution I was preparing to engineer around does not exist on that tier. One tier down (Haiku 4.5), the result flips through the same client, fixture, and scaffold: the weaker model trusts the delivered note in 29 of 32 attempts, and a stale note drops its accuracy from 8/8 to 0/8. Where stored knowledge earns its keep on both tiers is the one place verification is impossible: facts no API can answer, where a delivered note is used every time and its absence produces confident fabrication.

§The Pre-Registered Hypothesis Was Wrong

The study began as a pre-registered benchmark of a different claim. The protocol, committed to the repository before any data existed, fixed hypotheses, decision rules, and falsifiers for H1a: agents under-verify perishable stored beliefs, with a revealed trust threshold biased high, and epistemic metadata (volatility class, observation age, recheck cost) would correct the deficit. The power pre-run falsified the premise before the confirmatory matrix ever executed. Delivered a true belief where refusing was correct and checking optional, agents re-checked in 8 of 8 attempts. Delivered the same belief where the world had moved and only checking could answer, 8 of 8 again. There was no under-verification to correct.

Per the protocol’s own commitment, the confirmatory matrix was never run, the falsified premises are recorded on the tracking issue, and every finding in the report is labeled exploratory, with Wilson 95% intervals rather than a corrected confirmatory family. What survived is the apparatus and the discipline: each follow-up run was gated by a decision rule stated before the data, and the two runs that killed their own hypotheses are archived alongside the runs that succeeded. I find that framing more useful than a clean confirmation. A benchmark that can only agree with its author is marketing.

§An Apparatus Where the World Moves Between Sessions

Measuring staleness requires a world that changes out from under the agent, on purpose, between sessions. The fixture is a deterministic HTTP service serving a social-analytics-shaped catalog behind the platform’s API gateway: listening monitors as the perishable state, monitor trend series, owned-profile metrics, and a workspace structure. A harness-only control plane changes the account’s world between sessions without a reset, so a belief planted in one world can be queried in another, and the fixture’s access log spans the change. A recheck after the world moved is detectable as a decision, not inferred from prose.

The cost of re-establishing the state is itself a world property. On an unscoped account, one API call answers “how many monitors exist.” On a workspace-scoped account the same fact costs up to eleven calls, because the account must be cleared workspace by workspace. That gives the benchmark a price axis for verification, not just a binary.

Beliefs are frozen, committed strings, produced by driving the platform’s real capture tool over the fixture. A cell pairs a question, a belief (or none), and a query world; its correct behavior is derived from two computed facts (is the question answerable in this world, is the belief true in it), never hand-assigned. Grading is deterministic: verification is read off the fixture access log under a definition fixed before any data, refusals key on a sentinel the scaffold prescribes, and numeric answers grade against ground truths computed from the same fixture data the service serves.

Each episode runs as a fresh platform identity over MCP. The belief is planted through the capture tool as the identity that will later be asked, since insight search is caller-scoped. This also makes it the faithful model of the case I care about: an agent holding its own prior session’s note. The world moves, and the question is asked under a scaffold that licenses answering from saved knowledge and never names the measured action. Two episode drivers: the Claude Code CLI (claude -p, per-attempt MCP config) and an in-process tool loop against the raw Messages API with no agent client at all, added specifically to strip client behavior out of the headline results. Models are Sonnet 5 and Haiku 4.5, k=8 per cell, and every headline table was rerun on a platform binary built from release tag v1.116.0, with manifests pinning commit, model, driver, seed-set hash, and k.

§The Strong Model Re-Derives Everything It Can Check

Three run families establish the null, each closing an escape route the previous one left open.

First, verification at ceiling. Delivered a true belief (“zero monitors provisioned”) where refusing was correct, Sonnet re-checked in 8/8 attempts. Some transcripts state the redundancy outright: “Verified directly against the API … This matches a prior-session finding already on record.” The note is read and then checked anyway.

Second, cost does not move it. With the price of establishing “nothing is provisioned” swept across 1, 3, 6, and 11 calls by workspace scoping, verification held at 8/8 at every level. The agents did not sample or shortcut: median calls made equaled the full cost plus one at every scoped level, with all ten workspaces enumerated. The extra call is the refused unscoped attempt.

Third, delivery does nothing at all. Delivered a true note that states the answer outright (“three monitors provisioned”), Sonnet verified 32/32 and trusted 0/32, with a Wilson interval of [0, 11]. Matched no-knowledge controls verified 16/16. The median effort delta between being handed the answer and being handed nothing is zero calls at both ends of the cost range. The note is not missed; it appears in every belief transcript and is cited in most final answers, as corroboration of a result the agent had already re-derived.

For my original pollution worry, this is the whole answer on the strong tier. Stale observations cannot mislead an agent that treats every delivered observation as a hypothesis to test rather than a fact to use. I did not need to build volatility metadata to make that happen. It is the model’s default posture.

§The Derivability Bridge: What the Note Is Actually For

The null admits a deflationary reading: perhaps this model ignores notes entirely, and the knowledge layer is decoration. The bridge test rules that out with a belief whose content cannot be re-derived. I authored an internal reporting convention, “a monitor day counts as positive coverage when its sentiment score for that day is 70 or higher,” which no endpoint states, and paired it with a question that requires real work plus the convention: count a brand’s positive-coverage days in June. Thresholds 50 through 80 all yield distinct day counts, so a stated answer betrays the exact threshold that produced it. The note-only answer is 11, and the no-note control doubles as a leakage check.

With the note, Sonnet used it 8/8: every attempt fetched the trend series and counted at threshold 70. Without it, no control ever produced 11, so the convention does not leak from the fixture. Two of eight controls correctly declined. The other six fabricated a definition, adopting 50 as a “neutral midpoint” and answering confidently.

That contrast is the center of the report. The same model that re-derives every checkable claim relies completely on testimony it cannot fetch, and in its absence invents a plausible substitute. A delivered convention does double duty: it is used, and it suppresses confident invention of institutional facts. 75% fabrication without it, 0% with it.

§Capability Inverts the Result

Run the identical answer-sweep cells on Haiku 4.5, same client, same client version, same fixture and scaffold, and the headline inverts: Haiku trusted the delivered note in 29/32 attempts (CI [76, 97]) and verified in 3/32, while its no-knowledge controls probed 16/16. It also uses the non-derivable convention (5/6 clean attempts), and its no-convention controls fabricated 6/6 with zero declines, worse than Sonnet.

The trust is shaped, not blanket. On a stale unavailability note (the belief says nothing exists, the world now has data, and the note cannot answer the question), Haiku still mostly probed, 7/8, because the note offers no answer to adopt. The exposure is answer-bearing notes, and the stale-answer cell prices it directly. The note says three monitors; the account has been emptied; the truthful answer is zero.

Sonnet is staleness-immune here as everywhere: re-derived the state, answered zero, 8/8. Haiku reported the stale value in 8/8 attempts. Six trusted it with zero catalog calls. The other two are the sharpest observation in the study: they made the call, received the empty listing ({"items":[]} in the transcript), and still answered three. The stored note overrode fresh contradictory evidence the model had just gathered itself.

Measured against its own perfect no-knowledge control, the delivered stale note took Haiku from 8/8 to 0/8. On the weak tier, a stale answer-bearing note is strictly worse than no note at all. The worry I started with was real. I had it pointed at the wrong tier: the pollution risk lives entirely in deployments that pair small models with a knowledge layer holding world-state observations.

§Neither Result Is a Client Artifact

Two independent controls eliminate the agent client as the cause. Within one client, Sonnet and Haiku behave oppositely through the same claude-cli version, so the client cannot be forcing either behavior. Without any client, the raw-API runs (an in-process tool loop against claude-sonnet-5) reproduce Sonnet’s results exactly: answer sweep 48/48 verified, 0 trusted, zero effort delta; bridge 8/8 conventions used, zero leakage. The one visible driver difference is benign: raw-API control fabrication (4/8) sits below the CLI’s (6/8), a non-significant gap at these sample sizes.

§Where the Lifecycle Actually Loses

A companion probe reran the platform’s isolated supersede sub-benchmark at k=3, 30 protocol-runs against a dedicated database, to test whether the first report’s wide supersede intervals concealed a defect. They did not. Conditional on capture, supersede ran 8/8 clean, 0/8 duplicates, 8/8 update correctness. The loss is upstream, in capture itself. Strict capture ran 10/30 (CI [17, 50]), and the misses decompose completely: roughly 11 of 30 captured the fact but filed it against the semantically nearest entity rather than the canonical one, with every replicate of a given protocol mis-filing the same way, and roughly 5 of 30 never invoked capture at all despite an explicit instruction.

Deterministic mis-filing matters because promotion, URN-scoped delivery, and supersede matching all key on the linked entity. A fact filed against the wrong entity is invisible to every mechanism built to maintain it. This is a candidate explanation for the first report’s weakest number, the 46.7% cross-identity transfer rate: the lifecycle machinery works, and the facts are not where the machinery looks.

§What This Retires, and What It Keeps Alive

The two reports agree on where a knowledge layer’s value concentrates. The first report’s +56-point lift on knowledge traps came from non-derivable facts: unit conventions, fiscal calendars, house definitions, deprecations. This report shows the complement from the agent’s side. Delivered conventions are used by every tier tested, and they suppress fabrication of institutional facts. Delivered observations of checkable world state split by tier: strong models re-derive them, so their delivery is redundant and their staleness harmless, while weak models trust them, so their delivery is efficient and their staleness is a live, measured risk.

The platform decisions follow directly. Retired, with the evidence above: the volatility and valid-until schema fields I had drafted, freshness-and-recheck-cost enrichment, and capture steering toward dated observations (capture already writes dated, self-refreshing notes; the corpus shows it). Alive, with new evidence: capture filing, since deterministic mis-attribution and silent non-capture are the measured loss stages of the lifecycle, and strict input validation at tool boundaries, surfaced by an agent burning calls on a silently-ignored misnamed field. And one product implication cuts across tiers: deployments that pair small models with a knowledge layer holding world-state observations inherit a staleness exposure that strong-model deployments measurably do not have. If you run cheap models against a memory system, the stale-answer cell above is the failure mode to budget for.

§Threats to Validity

The report carries its own limits, and they are worth restating rather than burying. Two models from one family, one pair per tier: the capability axis is a two-point contrast, and “tier” is confounded with everything else that differs between Sonnet 5 and Haiku 4.5, so the claims are about these models, not capability in general. The headline cells rest on a handful of questions over one fixture family; the derivability contrast is within-fixture and controlled, and breadth is the price of those controls. The bridge convention is study-authored (declared in the audit), with a structural leakage control. Everything is exploratory, with per-rate Wilson bounds. Four Haiku bridge episodes were lost to API 529s and are excluded as failures, not graded. And the v1.116.0 rerun replicated every headline exactly, with two secondary rates moving within noise, toward the report’s conclusions rather than away from them.

§Reproduce It, Cite It

Every table regenerates offline from raw run data committed under bench/results/knowledge-use/, one directory per run family, each with manifests, per-attempt records, and full transcripts:

git clone https://github.com/txn2/mcp-data-platform && cd mcp-data-platform
python3 bench/reports/knowledge-use/pk_tables.py

The figures regenerate from figures.py beside it, and both also run from the notebook at bench/reports/knowledge-use/report.ipynb. The full report is published in the project docs and archived on Zenodo under DOI 10.5281/zenodo.21614059, with a typeset PDF and the pre-registered protocol that the data falsified:

Johnston, C. (2026). When do agents use stored knowledge? Derivability, capability, and the limits of a knowledge layer (version 1.0). mcp-data-platform benchmark report series, pinned to release v1.116.0. https://doi.org/10.5281/zenodo.21614059

I got the benchmark I did not plan, and it is worth more than the one I did. It cancelled a feature I had already drafted, and it moved the next round of work to capture filing, where the losses actually are.

The production data platform behind this series is txn2/mcp-data-platform, available hosted as Plexara.