What a scrambled poem
can teach us about AI.

Take a poem and shake it apart. Keep its words, but rearrange them. Shuffle its lines. Go further and scramble the small pieces of text that an artificial intelligence system actually reads.

For a human reader, the damage is obvious. Rhythm breaks. Sentences collapse. A declaration of love becomes a heap of verbal ingredients. But inside a language model, what survives?

That question motivated a research project at The Shore, conducted by me, with extensive AI assistance — like most of what ships here these days. It began with curiosity about how familiar literary texts are actually represented inside models whose internal calculations can be inspected.

The sonnets supplied a rich, manageable collection to explore. The choice of models supplied something equally essential: access.

Looking between the input and the answer

When people use a chatbot, they usually see what goes into the system and what comes out. They can ask questions, compare answers and introduce mistakes into a prompt. But those conversations do not ordinarily reveal the numerical representations produced as the model processes the text. This study required looking between the input and the answer.

So I used three open-weight models — Qwen from Alibaba, Muse from Meta, and Gemma from Google. Open-weight means the weights — the learned numerical parameters that define a model’s computations — are downloadable: you can run the model yourself and record what happens inside it as text moves through. I ran code that captured internal representations at different stages. That access made it possible to compare how the models represented an original poem, a shuffled poem and other poems in the collection.

To be clear about one thing: this study was only possible because of open-weight models. Access allows experiments that cannot be conducted through an ordinary chat interface alone. It does not make a model automatically understandable, but it gives you measurements you would otherwise lack.

Recognizing is not relating

The most intriguing finding was a separation between two things that can sound almost interchangeable: recognizing a text and preserving its relationships to other texts.

Imagine a library whose books have been scattered across the floor. You might identify each book from its vocabulary, even after its pages have been mixed up. But identifying the books would not automatically restore the shelves: which books belong together, which explore similar subjects, and which are unlike their neighbors.

The experiments examined both abilities. Could I match a scrambled poem’s internal representation to its original? And did the collection retain a similar arrangement — its map of similarities and differences — after scrambling?

The models were not asked to explain poetry. Instead, I recorded their internal representations: long lists of numbers produced while processing text. Comparing those lists can reveal which inputs a model distinguishes and which it places close together. Different ways of extracting and comparing those numbers can also produce different answers.

The research began with all 154 Shakespeare sonnets. Later experiments added separate sets of 24 sonnets by Philip Sidney and 24 by Edmund Spenser, with repeated randomizations and calculations at two numerical precisions.

A simple statistical adjustment

A particularly interesting result emerged from a simple statistical adjustment.

Scrambling text can move many poems’ representations in a common direction. That shared movement may obscure differences that remain between individual poems. I estimated separate average representations for original and scrambled inputs, then subtracted the corresponding averages before comparing them.

Crucially, I learned those averages from one poet and applied them to another. This tested whether the adjustment could work beyond the particular poems used to calculate it.

In one selected analysis of Spenser’s poems, using averages learned from Sidney, Qwen’s matching score rose from 6.3% to 83.3%. Muse’s rose from 18.8% to 93.8%, in comparisons involving sets of 24 candidate poems.

The adjustment did not repair the poems or teach the models anything new. It changed how I compared the models’ existing representations, exposing identifying information that had been difficult to detect.

The fingerprint alternative

There is also a straightforward alternative to the exciting interpretation: the scrambled inputs retained their original inventory of text pieces. A distinctive collection of words or word fragments could act like a fingerprint, even after the relationships between those pieces had been destroyed. Recognizing those ingredients would not establish that the poem’s meaning survived.

The second measurement sharpened that distinction. Improved identification did not generally restore the original map of relationships between poems. A representation could retain evidence of which poem it came from while occupying a changed position among the others.

The archive with imperfect scans

Why should anyone outside literary research care? Because many useful applications of AI depend on relationships between pieces of information. Document search depends partly on finding material related to a query. Systems that organize archives, group customer feedback or connect related passages depend on useful notions of similarity. In those settings, identifying a document and preserving its relationship to other documents are separate requirements.

Consider an archive containing imperfect scans. A damaged document might retain enough distinctive vocabulary to remain recognizable. A different question is whether a system still places it near the documents that would help a reader understand it.

This study did not test scanned archives or a production search system. Its relevance is that it provides a concrete reason to measure those outcomes separately. A strong result on one test need not settle the other.

The same distinction matters when evaluating models. Two systems might arrange intact texts differently yet respond similarly when those texts are disrupted. Different stages within one model may also retain different kinds of information. A single similarity score can conceal those differences.

The sonnets therefore serve as a test bed for a broader question.

What the study doesn’t claim

The report also shows why exploratory research needs restraint. Several predictions recorded before the later experiments failed to hold in full. An earlier finding involving agreement between Qwen and Muse did not transfer to the new poem populations. Those failures remain in the paper.

The promising identification results emerged during a later exploratory analysis, after outcomes for both additional poets were known. They deserve fresh tests with the analysis fixed in advance. The work has also not established that these measurements improve literary criticism: usefulness tests were mixed or negative, and no blinded human evaluations have been collected.

What the work offers is a sharper set of questions — and an illustration of why access matters.

Being able to inspect a model allowed me to look beyond fluent answers and examine how representations changed under controlled disruption. Public code, generation recipes and a compact verification dataset now let others check selected calculations and pursue those questions themselves.

The next challenge is to separate surviving vocabulary fingerprints from surviving meaning, and to test whether these distinctions matter in practical systems.

Shakespeare provided the starting material. The larger subject is how to tell what remains inside an AI when the information we give it changes.

The full paper and reproducibility materials are publicly available. The paper is a preprint and has not undergone independent peer review.