The two benchmark reports I published in July, the knowledge-layer ablation and the stored-knowledge study, both came out of the same piece of software: a benchmark harness in the bench/ directory of mcp-data-platform, roughly 36,000 lines of Go in its own module, kept out of the platform’s build on purpose. What I value most in that harness is turnaround. On July 31 I committed a probe instrument in the morning, ran a two-stage experiment across three models in the afternoon, and by the end of the day a study I had been considering was dead, killed by a decision rule stated before the first run. This post is about the machinery that makes an experiment cost an afternoon instead of a month.

First, the system under test. I have been developing mcp-data-platform in the open at txn2, and writing about it here since the platform build post in July. It is a Go server that sits between an agent and a data stack. Underneath, it composes the standalone txn2 toolkits for Trino, S3-compatible object storage, and DataHub behind one authenticated session. On top of those it adds the parts the benchmarks poke at: tool results automatically carry catalog context, a search tool fronts curated knowledge pages, and a capture-and-promote lifecycle lets a fact recorded in one session change how later sessions behave. Personas decide which tools an identity sees, and every tool call lands in an audit log. Those layers are what the benchmarks ablate, and the audit log turns out to be half the measurement story.

The whole module is thirty commits between July 12 and August 1. In that window it produced two reports with DOIs, killed three study candidates for cause, and re-ran the lifecycle suite at k=5 over a doubled protocol set. The first pilot commit landed July 12; the first DOI-archived report published July 19. Seven days from nothing to a citable result is amortization at work: fixtures, graders, drivers, and statistics are built once, and every subsequent question pays only for its own runs.

§Ablate the Platform, Not the Model

Model benchmarks ask which model is better. That is not my question. I ship a platform, and the claim worth testing is that the platform changes agent behavior. So the harness inverts the usual design: a run holds the model, the prompt scaffold, the seeded data, and the task set constant, and varies only the platform configuration.

The four arms of the knowledge-layer study are four YAML config profiles of the same running binary. Arm a0 exposes raw data tools with every semantic feature off. Arm a1 turns on cross-enrichment, so tool results carry catalog context automatically. Arm a2 adds the search tool and curated knowledge pages. Arm a3 adds the memory and knowledge-capture lifecycle. Nothing else moves. The system prompt is a shared constant in the pipeline code, identical across arms, because the moment the prompt varies with the arm, the study measures prompt engineering.

Config-as-ablation has a second payoff that took me a while to appreciate: new experiments are config files. The gate probe that died in a day needed one new arm, a single-deviation copy of an existing profile with workflow.require_search flipped off. The expensive parts, the fixture, the cells, the analyzer, already existed.

The rest of this post walks down the layers of the apparatus, so here is the whole of it in one place:

§Three Drivers, One Transcript Model

The harness runs agents three ways, and all three produce the same transcript format.

The canonical driver is the raw Anthropic Messages API with the harness’s own tool loop: no agent client, no client-side behavior, every tool call visible. Each task carries a tool-call budget (15 to 30 in the committed tasks) plus a two-iteration wind-down, so a model that burns its budget in multi-call turns cannot spin forever. Budget-refused calls return a distinguishable sentinel, so analysis can tell “tried and was refused” from “never tried.”

The second driver invokes headless Claude Code, claude -p, with a generated per-attempt MCP config. This is the realistic client path, and it is subscription-funded, which matters for cost (more below). The harness disallows Bash, the file tools, the web tools, and subagents for it, and each attempt runs in a fresh empty temp directory so the model has no repository context to lean on. The two paths are never merged into one leaderboard. Every manifest records the client version, and the merge tool refuses to fold an API pass into a CLI pass, because a Claude Code update can move numbers that have nothing to do with the platform.

The third driver is scripted playback: a JSON file of canned model turns. It is the reason every grader and every metric has a deterministic end-to-end test. make bench-smoke exercises the entire pipeline, fixture reset, session minting, tool dispatch, audit read-back, grading, and metrics, with no API key and no cost.

The reason there are two model-facing drivers is a lesson from the study that died in July. In the API-connection pilot, the arm that was supposed to represent “all tools in context, no search” turned out to measure something else entirely under Claude Code, because the client applies its own tool search over large toolsets. The naive condition I thought I was running never ran. Client behavior is a confound you strip by having a no-client path, and I did not fully believe that until it bit me.

§Ground Truth Is Generated, Never Typed

Every task answer in the suite is computed, not written. One fixed-seed generator (seed 930) emits the entire experimental world: the warehouse DDL and rows, the catalog metadata, the knowledge pages, the 87-task suite, the 30 lifecycle protocols, and the 6-lesson cold-start curriculum. Ground truths are derived from the generated rows by the same code that generated them. If a trap invariant fails, say the gross-revenue leader and the net-revenue leader come out identical, so the net-revenue trap no longer discriminates, generation panics. The seed is a constant, so a violation is a code bug, never a bad roll.

Committed artifacts are pinned by drift tests: regeneration is byte-identical, and a test diffs the committed files against a fresh run. Change the generator and the committed tasks in different commits, and CI fails. Every run manifest records the task-set hash, so a result can never be quietly compared against a different suite than the one it ran.

Hand-typed answers rot. Generated answers cannot drift from the data, because they are the data.

§The Audit Log Is the Measuring Device

The subtlest design choice in the harness is what counts as an observation. Transcripts lie by omission: a tool call that fails at the transport layer may or may not have reached the platform. So the harness treats the platform’s own audit log as the instrument. Arm profiles run audit in synchronous delivery, meaning a returned tool call has a committed audit row, and after every attempt the harness reads back the session’s rows by session id.

The accounting is three-way: calls confirmed audited, calls refused before the audit middleware (gate and authorization refusals leave no row, by design), and transport failures where the platform may have audited before the error surfaced. The run computes bounds from its own client-side accounting, and if the audit rows fall outside those bounds, the run fails loudly. The protocol document states the rule plainly: a run with unaccounted attempts is not publishable.

The same instrument thinking shows up in the stored-knowledge study, where the measured behavior is verification. Whether an agent re-checked a belief is read off the fixture’s access log under a definition committed before any data existed, never inferred from what the transcript says it did.

§Identities Are a Control Variable

The platform’s search-first gate keys discovery on the authenticated user. Run every attempt as one shared credential and the first attempt’s search opens the gate for all the rest, which silently contaminates every later episode. So the harness maintains a pool of 320 committed identities, every attempt authenticates as its own, and lifecycle protocols consume two per attempt, a teacher and a learner, so a fact taught in one scope can only reach the learner through the platform’s own delivery machinery. A run refuses to start if tasks times k exceeds the pool. It also refuses to start against identities that already hold insights, after a pilot where agents found their own earlier notes, correctly declined to duplicate them, and the run archived empty capture episodes that looked like failures.

§Grading Is Deterministic First, and the Judge Is Confined

Five grader kinds, all deterministic, all borrowed from named benchmarks rather than invented here: numeric tolerance matching on the first line after the final answer marker; entity matching with a wrong-alias veto, so mentioning the trap answer fails the attempt even alongside the right one; SQL graded by execution-result comparison in the BIRD convention, where two different but equivalent queries both pass; post-run fixture state checks in the MCPMark style; and refusal grading in the BFCL irrelevance style, where a pass requires both declining and making no write call, with writes read from the fixture access log.

One axis cannot be graded deterministically: whether an answer carries a required caveat. That gets an LLM judge, and the judge is kept in a small cage. Its model is pinned in the rubric file, the rubric is versioned, and it ships with a 30-item human-labeled calibration set, deliberately balanced so agreement cannot be an artifact of a one-sided prior. The rubric header cites the cautionary tale that motivated all of this: MCP-Atlas, a published benchmark whose scores moved because a judge model upgraded silently underneath it. The judge never touches a headline number in either report.

§Statistics With a Fixed Seed

Uncertainty is a percentile bootstrap with a fixed resampling seed, so the confidence intervals themselves reproduce byte-for-byte. Repetition is k attempts per task with pass^k in the tau-bench convention, and harness failures are excluded from accuracy and reported separately, never counted as model errors. One shared rate type carries every metric in every suite, so there is no per-suite fork that could drift on how an empty denominator is handled.

The protocol document also holds something I had never written into a project doc before: an a-priori power calculation working out the smallest resolvable effect at each denominator. It concluded the lifecycle suite could not resolve the effects worth claiming at fifteen protocols, and the suite was doubled to thirty and the identity pool grown to 320 to make k=5 reachable. That analysis paid off on August 1, when the k=5 rerun turned a duplicate-rate interval 72 points wide into one 24 points wide.

§Never Destroy Paid-For Evidence

Model runs cost real money, and the harness treats their output the way a lab treats samples. Results flush to disk after every attempt, so an interrupted run keeps everything it paid for. Runners refuse an output path that already exists, with no override flag; the operator picks a new path. The merge tool compares files by device and inode rather than by path string, specifically to catch case-variant paths on macOS clobbering an input with its own output. This rule exists because early adapter-validation runs reused transcript filenames during development, and that data is gone.

The archive policy is the same discipline at the directory level: every run that produced output is kept verbatim, including the defective, the interrupted, and the uninteresting, each with a README stating what it does and does not establish. The results tree currently holds about 105 MB of raw run data, and both reports recompute every published number from it offline, no API key, no network.

§What It Costs

The harness is aggressive about cost because the alternative is not running experiments. The Anthropic driver sets two prompt-cache breakpoints, one on the constant tools-and-system block and one rolling with the transcript prefix, and records cache token counts in every attempt, so each results file documents its own cost basis. The Claude Code driver runs on a subscription, no metered key, which is why broad probes default to it and the raw API is reserved for the runs where client behavior must be stripped out. The k=5 lifecycle rerun, five full passes with platform resets between them, cost about twenty dollars. The judge is pinned to a mid-tier model on cost grounds, stated in the rubric. And the findings register prices deferred studies in dollars as an explicit reason for deferring them.

§What Ports and What Does Not

The harness is coupled to mcp-data-platform where it reads the platform’s admin APIs: audit events, insight and changeset state, session minting. The rest ports. The agent loop, the three drivers, the grader kinds, the caged judge, the bootstrap statistics, the identity pool, and the report model are generic MCP-facing machinery. One piece is generic enough to reuse tomorrow: epmcp, a server that takes any OpenAPI spec and exposes one MCP tool per operation, built to represent the market-default gateway pattern. Pointing it at a 2,503-operation catalog is how the harness measured that pattern costing 365,902 tokens of tool definitions against a fixed 3.2k-token search surface, at equal accuracy.

The most portable part is the method: hold the model constant and ablate your system through config; generate ground truth instead of typing it; measure from logs, not transcripts; grade deterministically and cage the judge; fix the bootstrap seed; and never delete a run. None of that is specific to my platform, and all of it is why the next question costs an afternoon.

What the instrument gets used for, including the three studies it killed, is the next post.

The production data platform behind this series is txn2/mcp-data-platform, available hosted as Plexara.