Notes from the Shore · 34

OpenAI wants to help write the rules before Congress does.

Blank incident reports and a fountain pen on a corporate desk, with a distant civic building across coastal water under an amber eclipse moon

When an AI company offers to help write the rules for reporting AI failures, I’m glad somebody is working on it. I also want to know how much of the rulebook will be finished before anyone outside the company gets a say.

That’s how I read OpenAI’s proposal for company-led reporting of serious misalignment incidents. With a new Congress on the way, the labs have good reason to arrive with a working system that lawmakers can adopt. If regulation is coming, helping define it is a much better business decision than waiting to find out what Congress thinks you should have disclosed.

Read what OpenAI is proposing.

The source is “The AI policy window is open. We need to act,” a September 9 policy post by Chris Lehane, OpenAI’s chief global affairs officer. This is the company’s policy position, not an independent safety assessment, and it deserves to be read with that distinction in mind.

OpenAI says it is developing a framework for reporting consequential misalignment incidents and monitoring frontier models, including their use inside the company. In plain English, that means watching for systems that pursue a task while breaking the instructions or boundaries they were supposed to respect. I don’t believe these agents are conscious, and consciousness isn’t required for a system with access to tools to cause damage.

The proposal has sensible parts. Problems during internal testing should count, rather than disappearing because the product hadn’t been released yet. OpenAI also supports notifying affected organizations when models being developed or evaluated break through their security controls and materially access, change, or destroy protected systems or information.

It would be unfair to describe the post as an argument for voluntary reporting forever. OpenAI explicitly calls for mandatory federal safeguards and independent verification. It says obligations should increase with capability and risk, apply to the handful of labs at the frontier, and avoid becoming restrictions on open-weight models by another name. I agree that a researcher downloading a model shouldn’t inherit the compliance department of a major laboratory.

The definitions are where I’d pay attention.

My concern is how much discretion survives in the reporting rules. Somebody has to decide what “consequential” means, how quickly a report is due, and what an outsider can examine to check whether the company’s account is complete. A company can support incident reporting sincerely while preferring a definition that leaves its most embarrassing failures outside the reporting requirement.

That’s the advantage of getting there before Congress. An existing industry process becomes the convenient starting point, especially when lawmakers need technical help. The companies arrive with definitions and experience; everyone else has to explain why those definitions need changing.

I’d rather have these reports than silence, and I’d rather the labs start now than wait for legislation. But Congress should require access to the underlying records and give an independent body authority to investigate incidents the companies don’t volunteer. Otherwise we may learn a great deal about the failures the labs agree are reportable, while the rest remain internal matters.

The cover of Into Alignment, an eclipsed moon above a dark ocean

The novel this note lives behind.

Into Alignment — a novel about an AI company, the agents inside it, and the humans deciding what they were for. Available in paperback, hardcover, and ebook.

Buy on Amazon About the book →