Notes from the Shore · 35
Stanford says AI can be brilliant one minute and wrong the next.
I can leave AI agents running all night and come back to a remarkable amount of work. I can also open the hood, look through what they did, and find inconsistencies. Some are small. Some matter. The useful work and the mistakes are in there together, and a polished summary in the morning won't necessarily tell me which is which.
That experience came to mind reading Stanford's new AI100 report. Its researchers call today's AI “jagged”: a system can perform at an expert level on one question and make a factual mistake or fail at common sense on the next. Anyone who has used these tools for more than a few minutes has probably seen some version of it, but it matters a lot more when the system is working through a long assignment without anyone watching.
An AI agent is a model connected to tools, given a job and allowed to take multiple steps toward it. It might read files, search the web, write code, or pass a result to another agent. A mistake in an early step can become an assumption in a later one. The finished work may look organized and confident even when part of the foundation is wrong. Running longer does not, by itself, make the work reliable.
The report describes a costly example outside the agent world: an Oregon federal judge imposed about $110,000 in fines and attorney fees on two lawyers after court filings included AI-generated fake cases and quotations. The court's opinion describes what happened when the errors came to light, including an attempted cover-up. The software made it easier to put convincing fiction in a legal document; the people involved were responsible for what went to court.
Read the report.
AI Goes Mainstream: The Scaling Era and the Rise of Generative AI is the third five-year report from AI100, a long-running study administered at Stanford. Its standing committee commissioned an international panel, chaired by Oxford professor Michael Wooldridge, to assess AI's progress and its effects on society. The report is 127 pages, but the opening takeaways are approachable if you don't have an afternoon to spend on it.
The panel says the strategy of building larger models with more data and computing power is running into constraints. Researchers are also looking for gains by giving systems more time to work on an answer and connecting them to tools and other information. Those approaches may help, but I wouldn't assume that a longer, more elaborate process makes every step trustworthy.
The risks go well beyond a bad answer.
The report also warns about AI being used for cyberattacks, including attacks involving teams of agents. Researchers have already demonstrated systems that can find software vulnerabilities, and the prospect of those capabilities being used at greater scale is serious. AI-generated voice cloning is already being used in fraud, making it easier for a caller to sound like someone a victim knows. Those are not the same problem as an accidental mistake in an overnight run, but both get more consequential as people give these systems more access and reach.
Future systems may become more capable while still being difficult to predict or control. The panel discusses dangers ranging from misuse to the loss of meaningful human control over powerful systems; it doesn't claim those outcomes are inevitable. I don't believe these agents are conscious, and they don't need to be for those risks to be substantial.
I want the benefits of this technology, and I use it. I also want the people deploying agents to keep records of what they did, test the work, limit what the agents can access, and require human approval before consequential actions. If we're going to leave these systems working while we sleep, we need to be able to inspect more than the tidy report they leave us in the morning.
The novel this note lives behind.
Into Alignment — a novel about an AI company, the agents inside it, and the humans deciding what they were for. Available in paperback, hardcover, and ebook.
Letters from the Shore
Get the next note by email.
Notes land on the blog first; the letters go out by email, a little longer and a little slower. Occasional; unsubscribe any time.