A point in four thousand dimensions.

If yesterday’s note lost you, this one is for you. It was a technical post — I scrambled Shakespeare’s sonnets, fed them through three open-weight models, and recorded what the models kept. But the experiment only makes sense if you know what a language model is doing between the words going in and the answer coming out. So here is the machinery, as plainly as I can put it.

It starts with tokens

When you type something into a chatbot, the model does not read words the way you do. First the text is cut into tokens — chunks of text. Sometimes a token is a whole word. Sometimes it’s a piece of a word; a long name might come apart into two or three pieces. Each token is basically a vocabulary entry with an ID number.

Then something less obvious happens: every token is turned into a list of numbers. That list is where the real work starts.

A vector is a list of numbers

The list is called a vector, and the word sounds more intimidating than the idea is. A vector is a list of numbers that locates something.

Think about points in space. To describe a spot in a room, three numbers will do it — how far along, how far across, how far up. Three numbers, three dimensions, and every point in the room has its own address. Navigation is the same trick with latitude, longitude, and altitude. That’s a three-dimensional vector: coordinates for a point.

A language model does the same thing, except instead of three numbers it uses a great many more. When a token enters the model, it becomes a point in a space with thousands of dimensions — fifteen hundred in some models, four thousand in others. Nobody can draw that space, and I can’t picture it either. But the mathematics doesn’t need a picture. What the dimensions carry is meaning-by-neighborhood: tokens with similar jobs end up close together, and unrelated ones land far apart.

Tensors, and the levels they move through

Now it gets a little more complex, because the tokens don’t jump from vectors straight to answers. They pass through a series of systems.

First, the word tensor. A tensor is an arrangement of numbers with more than one direction. A single list is a one-dimensional tensor. A spreadsheet — rows and columns — is two-dimensional. Stack the spreadsheets and you have a three-dimensional tensor, and it keeps going from there. Nothing mystical about the word: a tensor is a container for numbers, with enough directions to hold whatever the model is juggling at once. The tokenized text, as a bundle of those multidimensional vectors, gets wrapped into tensors.

Those tensors then move through the large language model itself, which is built on something called a transformer architecture. Picture a stack of levels. The text enters at the bottom and climbs. At each level, every token looks around at every other token — that part is called attention — and the numbers get adjusted, some of the context of the whole passage folded into each token’s coordinates. The models I work with have dozens of these levels; forty-eight is not unusual. What comes out the top is the model’s prediction for what comes next.

What I measured

Which brings me back to yesterday. As the sonnets climbed through those levels, I recorded the numbers the model was holding internally at each one — the internal mathematical representation, what researchers call the activations. Then I scrambled the poems and recorded again.

The comparison across all the levels is the study. Did a scrambled poem still sit near its original — did the poem’s location in that huge space survive the scrambling? And did its relationships survive: the distances between one sonnet and the next, the arrangement of the whole collection. The short version of the finding: the poems stayed identifiable, but their arrangement among the other poems mostly did not come back. The long version, with the chart and the honest caveats, is yesterday’s note.

None of which means the model understood the poems. Knowing where a text belongs is not the same as having read it. Even if you think these LLMs understand what you’re asking them to do, they don’t — and some days I’m not sure people do either.

A lot of training, a little bit of math, and some chance — and underneath all of it, geometry. I’m not sure I’m able to explain it so most people can understand. I’ll keep trying.

The paper behind all of this, Recoverable Poem Identity and Changing Relational Geometry Across Transformer Depth, is free to read on Zenodo. It’s a preprint and has not undergone independent peer review.