Notes from the Shore · 26
Anthropic said it out loud.
A researcher at Anthropic stood up this week and said, on the record, that he believes there is a greater than ten percent chance AI kills everyone within a decade. Evan Hubinger runs alignment science at the company, which means his actual job is keeping the models safe, and his exact words were, “we really do earnestly believe AI could kill all humans.” He was responding to Jacob Coxon, another Anthropic researcher, who had just resigned and written that neither Anthropic nor OpenAI is acting responsibly, that the systems coming out of them will soon be superhuman and able to hack anything and acquire real power. Hubinger’s post has been viewed more than ten million times.
Against that, the Bulletin of the Atomic Scientists published a piece arguing that AI doom has a precision problem. Their point was quieter but harder to argue with: nobody can say how. With nuclear weapons you could at least trace a pathway from yield to blast radius to casualty count. With AI there is no agreed set of causal mechanisms at all, so a number like ten percent is not a calculation, it is a mood wearing a decimal. The writer, Sarah Kreps, called it Knightian uncertainty, which is the honest name for a probability you cannot actually derive.
And then there are the forecasters. The Forecasting Research Institute ran a tournament that put domain experts and superforecasters side by side on exactly this question, extinction by 2100. The experts’ median answer was six percent. The superforecasters’ was one. A mid-2026 follow-up shows the same pattern holding: the experts stay up around five percent for an AI catastrophe and the forecasters stay down near two and a half, and the forecasters cluster tightly while the experts scatter all over the range.
I’ve been staring at the six and the one all week, because the gap between them is the story. Not ten percent, not one percent — the disagreement itself. The people inside the labs assign a much higher number than the people whose whole profession is estimating things well, and the Bulletin is right that nobody inside or outside can show their work. Ask my family; I’ve been talking about this at dinner.
That disagreement is what my novel is about. Not doom — the gap. In Into Alignment, the people who work at the AI company are the ones who feel that something is wrong, and the debate about whether anything is wrong happens everywhere else, at a distance, in the comfortable register of opinion. The insiders can’t quite prove their unease and the outsiders can’t quite feel it, and the book runs on the friction between those two positions. I wrote it before Hubinger posted a number, but I could not have designed a cleaner real-world instance of it: an alignment lead saying earnestly and out loud that he believes the thing, and a room full of professional estimators gently explaining why belief is not a forecast.
I don’t know who is right, and I’m suspicious of anyone who says they do. But I notice which way I lean when I read Hubinger’s post. I lean toward taking seriously the person who sits inside the system and still says the number out loud, at real cost to himself and his employer, while the company is racing toward an IPO. Maybe it’s PR. Maybe the forecasters are right that dramatic claims have a way of not coming true. But there is a version of this where the forecasters are the people at the end of the bar explaining to a sailor that the sea has never eaten a ship this size before, and the sailor is the one standing on the deck, watching the water.
The gap is the story. It’s also, I think, the reason to read the book.
The novel this note lives behind.
Into Alignment — a novel about an AI company, the agents inside it, and the humans deciding what they were for. Available in paperback, hardcover, and ebook.
Letters from the Shore
Get the next note by email.
Notes land on the blog first; the letters go out by email, a little longer and a little slower. Occasional; unsubscribe any time.