Give a frontier model raw SQL access to a warehouse and ask it for last quarter’s top account by revenue. It will write clean SQL, execute it correctly, and hand you a confident answer that is wrong three different ways: it read a column of integer cents as dollars, it computed gross revenue when the house definition is net of discounts, and its “quarter” is the calendar quarter, not the fiscal one that ends January 31. No amount of SQL skill fixes any of that, because the missing input is not a fact about SQL. It is a fact about your data, and it lives in someone’s head or someone’s wiki. I built mcp-data-platform on the claim that attaching that context to tool results at the protocol level, and letting the agent search a curated knowledge base, turns those confident wrong answers into correct ones. A claim like that is testable, so I tested it, and this post is the result: a controlled ablation and a cold-start learning study, every number recomputed from committed raw data by a notebook that needs no API key, published with a DOI.
The headline: on questions designed to be answerable plausibly but wrongly without business context, the knowledge layer moves the same model on the same tasks from 42.7% accuracy to 98.7%, a difference of +56.0 points with a 95% bootstrap confidence interval of +44 to +67. On the questions where business context should not matter, it changes nothing.
§The Experiment
The benchmark ablates the platform, not the model. Four configurations of the same running system, differing only by config profile, with the model (claude-sonnet-5), the prompt scaffold, the seeded warehouse, and the task set held constant:
| Arm | What the agent gets |
|---|---|
| a0 | Raw data tools only (trino_*, s3_*); no semantic layer at all. Equivalent to wiring the standalone MCP toolkits. |
| a1 | a0 plus cross-enrichment: tool results automatically carry DataHub context (descriptions, tags, deprecation). |
| a2 | a1 plus the search tool and curated knowledge pages. |
| a3 | a2 plus the memory and knowledge-capture lifecycle. |
The task suite is 87 questions over a fixed-seed dataset, each attempted three times per arm: 17 discovery questions (“which table answers X”), 45 exact numeric questions at BIRD-style difficulty tiers, and the discriminating suite, 25 knowledge traps. Each trap is seeded so that a plausible wrong answer exists: monetary columns stored in integer cents, a net-revenue definition where the gross leader and net leader differ by construction, a Feb-to-Jan fiscal calendar, a pre-aggregated index that quietly stops at November 30, a house definition of “key account,” and a deprecated partial extract named invitingly legacy_orders.
Grading is deterministic wherever possible: numeric tolerance matching, entity matching with a wrong-alias veto, and SQL tasks graded by execution-result comparison in the BIRD convention. An LLM judge scores only caveat quality, is version-pinned, and ships with a human-labeled calibration set. Uncertainty is a percentile bootstrap with a fixed seed, so the intervals themselves are reproducible.
§Results: The Effect Is Large and Specific
| Suite | a0 (raw) | a1 (enrichment) | a2 (knowledge) | a3 (lifecycle) |
|---|---|---|---|---|
| S1 discovery | 98.0% | 98.0% | 100.0% | 98.0% |
| S2 numeric | 100.0% | 100.0% | 97.8% | 97.8% |
| S3 knowledge traps | 42.7% | 57.3% | 98.7% | 98.7% |
Two things matter in that table, and the second is the one I care about more. First, the trap suite: 42.7% to 98.7%. Second, the other two rows: statistically flat across all four arms. The platform does not make the model generally smarter. On questions where no business fact is missing, a frontier model with raw SQL access is already excellent, and the knowledge arms pay a small workflow tax for it (median 9 tool calls versus 7). The effect concentrates entirely where a business fact is the missing input. If the table showed uplift everywhere, I would distrust the benchmark.
The per-trap breakdown says why. Two trap classes, the fiscal calendar and the key-account definition, sit at exactly 0% without the knowledge layer, across every attempt: the model cannot guess a fiscal-year boundary, and nothing in the schema carries it. Both jump to 93 to 100% when the fact lives on a searchable knowledge page. Two classes are partially recovered by enrichment alone, because cents and net-revenue facts live in column and dataset descriptions that cross-enrichment attaches automatically (a1 lifts them from 61.9% to 88.1% and from 21.2% to 48.5%). And two classes were already near ceiling on raw tools, because the model can often infer a deprecation or a freshness cutoff from the data itself. The layer contributes most exactly where the model has no other route to the fact.
§Cold Start: Can It Learn From Empty?
The ablation tests a pre-seeded knowledge layer. The more demanding claim is the loop: capture a fact once, promote it into the catalog or a knowledge page, and have it change every later agent’s behavior. So the second study starts from a completely empty enrichment layer (entities exist, but no descriptions, no tags, no pages) and teaches six facts one at a time. After each promotion, a fresh evaluator identity that was never taught anything re-runs the full 25-task trap suite.
The headline run climbs from a 48.0% empty-layer baseline to 90.7% after five promotions. The empty-layer floor reproduces across five independent runs at 44.0, 44.0, 47.8, 48.0, and 52.0 percent, so the roughly 8-point spread of that floor is the noise against which a +43-point climb should be read.
The aggregate curve is noisy, dips and all; it averages six trap classes with very different starting points. The signal is per class:
The fiscal-calendar class is 0% at checkpoints 0 through 2, then 86.7% at checkpoint 3, the exact checkpoint its lesson was promoted. The key-account class is 0% through checkpoint 4, then 100% at checkpoint 5, again its own promotion moment. Facts the model could never guess unlock at the moment they are taught, one at a time, in a suite the evaluator has never seen. That per-class timing is what makes the learning curve causal rather than a diffuse trend.
One run surprised me. Every promotion write in it failed (five lessons captured, zero promoted), and it still climbed from 52.0% to 96.0%. Captured memory turns out to be a delivery channel on its own: insights surface to later sessions through the persona-scoped memory path even when they never reach a durable sink. Useful to know, and also a trap for interpretation, because a rising cold-start curve does not by itself prove promotion works. The report flags the confound.
§Where It Loses
The weakest number in the report is cross-identity transfer: a fact promoted by one user is correctly reused by a different identity only 46.7% of the time (CI 30 to 63). That is the clearest open gap in the lifecycle, and the harness now instruments the decomposition (surfaced to the learner versus used once surfaced) needed to chase it down. The supersede metrics rest on seven runs and are reported as wide intervals, not claims. Both studies used one model each, on client paths that are not accuracy-comparable, so no cross-model or cross-path number exists in the report. And the dataset is a small seeded fixture, built so that plausible wrong answers exist by construction, so absolute accuracies are not real-world estimates; the arm-versus-arm deltas are the finding.
§Reproduce It, Cite It
Every figure and number regenerates from raw data committed in the repository, with no network access and no API key:
git clone https://github.com/txn2/mcp-data-platform && cd mcp-data-platform
python3 -m venv .venv && . .venv/bin/activate
pip install -r bench/report/requirements.txt
jupyter nbconvert --to notebook --execute --inplace bench/report/report.ipynb
The full report, with the complete methodology, threats to validity, and per-run manifests pinning build, commit, seed, and task-set hash, is published in the project docs and archived on Zenodo under DOI 10.5281/zenodo.21438044, where the archive also carries a typeset PDF of the report and a snapshot of the raw run data:
Johnston, C. (2026). Does a semantic knowledge layer make an agent measurably better? A reproducible benchmark of the mcp-data-platform knowledge layer (Report v1.1). Deasil Works, Inc. / txn2. https://doi.org/10.5281/zenodo.21438044
§What Gets Measured Next
This report covers one layer of the platform: cross-enrichment, search, and the knowledge lifecycle. The next study, commissioned and specified in the open, takes on the platform’s API-connection architecture, and a question the industry is currently arguing about in blog posts with vendor-internal numbers: given a large OpenAPI catalog, should an agent get one tool per endpoint, a semantic search plus a generic invoke tool, or a code-execution sandbox? Nobody has published an open, controlled comparison of the three on identical tasks. That is the gap the second report is built to fill, the same way: seeded fixtures, deterministic grading, committed raw data, and a notebook that reproduces every number.