Professional notes by Craig Johnston
long-form, short-form, working drafts · since 2008
long-form, short-form, working drafts · since 2008
VOL. XIX · MMXXVI
140 NOTES IN PRINT
140 NOTES IN PRINT
Tag: Benchmark
CXXXV
An Instrument for MCP Agent Studies
The harness behind two DOI-archived benchmark reports, and why a study premise can be killed in one working day
2026-08-01
CXXXIWhen Do Agents Use Stored Knowledge?
A benchmark that killed its own hypothesis: strong models re-derive what they can check, and a weak model trusts a stale note over the evidence in front of it
2026-07-26
CXXVIIIDoes a Semantic Knowledge Layer Make an Agent Measurably Better?
A reproducible benchmark: 42.7% to 98.7% on knowledge-trap questions, and a platform that learns from empty
2026-07-22