Notes from the Shore · 27
Wait, shouldn’t we be able to see the numbers?
On November 5, METR published an independent assessment of Anthropic’s internal AI research, and the sentence that stopped me was METR’s own, sitting near the top of the report: “the public has to rely on METR’s judgment much more than would be ideal.” Four researchers spent most of August through October inside the company’s data, with more access than any outside group has had, and they came out the other side telling us to take their word for it. That is not the shape a transparency-minded industry promised us. Ask my family; I have been saying this at dinner all week.
The findings themselves are stark enough. METR’s team reviewed internal benchmark scores, compute metrics, forecasts for unreleased models, and interviews with Anthropic staff, and concluded that Anthropic is likely improving the capabilities of its models on well-specified tasks at around twice the rate it managed between March 2024 and October 2025. They attach high uncertainty to the number and say they wouldn’t be surprised by anything from 1.3x to 3x. On one internal benchmark, the trend implies the company made roughly two years of historical progress in the last year. A majority of that acceleration is likely attributable to Anthropic using its own AI to do the research, which is the recursive part everyone has been waiting for. And they saw no significant bottlenecks ahead, in algorithmic progress, compute, or data, over the next twelve months.
Anthropic, for its part, had unilateral rights to redact non-public information from the report, and it used them to remove a majority of the evidence METR says it found most informative, on the stated grounds that the details could accelerate competing AI developers. The benchmark figures METR could publish point one direction, but the internal analyses that actually convinced the team, the attribution estimates and the material suggesting the acceleration may be higher than 2x and increasing, are blacked out. What the public gets is the verdict without the exhibits.
Wait. Shouldn’t we be able to see the numbers?
This is the most load-bearing measurement question of the decade, whether AI is starting to accelerate its own development, and the answer is being delivered as a trust exercise. I keep turning over the recursion in it: the report says AI tools did much of the accelerating, and the evidence for that claim is withheld because seeing it might accelerate someone else’s AI tools. The reasoning is coherent. It is also the sound of a safety case eating itself.
A good deal of my novel lives in exactly this gap. In Into Alignment, the people outside the company argue about the forecasts while the people inside hold numbers no one else has seen, and the story runs on what that asymmetry does to both groups. I wrote it before any of this arrived, but I recognized the shape immediately: the farther you stand from the building, the more you are asked to accept on faith.
I want to be fair here. Anthropic commissioned the assessment at all, which is more than most labs have done, and METR describes the access as surpassing its expectations. The competitive-secrecy argument is at least a real argument rather than a fig leaf, since frontier research pipelines are genuinely worth stealing. But a report that says the trend has doubled and the proof is secret is not accountability; it is a press release with footnotes. If the pace really is doubling and compounding, the public is being asked to trust an accelerating system on evidence it cannot check, indefinitely. So, genuinely: shouldn’t we be able to see the numbers?
The novel this note lives behind.
Into Alignment — a novel about an AI company, the agents inside it, and the humans deciding what they were for. Available in paperback, hardcover, and ebook.
Letters from the Shore
Get the next note by email.
Notes land on the blog first; the letters go out by email, a little longer and a little slower. Occasional; unsubscribe any time.