2. What a language model learns

Imagine an ordinary support ticket: someone bought one desk fan, sees two charges on a card statement, and wants the money back before rent comes due. The complaint brings together facts, feelings, rules, and institutional incentives—the same collision that gives Walt’s work its weight.

A useful response requires several kinds of competence. The assistant must recognize the complaint, distinguish the customer’s account from a verified transaction, understand the urgency, discover what the company permits, and explain what can happen next. A large language model (LLM) supplies part of that competence. Whether money actually moves depends on other software and the authority attached to it.

Tokens, weights, and pretraining

A language model receives text through a tokenizer, which converts it into a sequence of numbered units called tokens. A token can represent a word, part of a word, punctuation, or another text fragment; the division depends on the tokenizer. Breaking text into smaller units helps a system represent unfamiliar names and words without requiring a separate vocabulary entry for every possibility. Sennrich, Haddow, and Birch demonstrated the usefulness of subword units for handling rare words in machine translation in their 2015 paper. Modern tokenization has several variants; their method is one influential foundation.¹

Our ticket becomes such a sequence. Its significance is not supplied by the token numbers themselves. The model has learned numerical relationships that allow it to transform those inputs into representations useful for predicting what comes next.

Those learned numerical settings are the model’s parameters, often called its weights. During pretraining, a model processes large amounts of training material. For a GPT-style language model, a central task is predicting the next token from the preceding text. Training adjusts the weights to improve those predictions across many examples. During ordinary use, or inference, the model applies its learned weights to the supplied context and produces a distribution of possible next tokens. A selection procedure chooses a token, and the process continues. It need not always choose the single most probable token. OpenAI’s 2019 GPT-2 report is an early large-scale demonstration of this approach.²

The broad training task gives the system reasons to learn grammar, factual associations, styles of explanation, and patterns that connect situations with their consequences. To continue a passage about a disputed purchase plausibly, it helps to represent who paid, what was bought, and which claim remains disputed. Predicting language can therefore demand considerably more than matching adjacent words.

Training does not supply a complete, searchable copy of everything the model encountered. Nevertheless, memorization is possible. Carlini and colleagues demonstrated that carefully chosen queries could elicit verbatim training passages from GPT-2. Their finding establishes a real extraction risk for the model and conditions studied; it does not mean every passage is recoverable from every model.³

For the fan complaint, broad familiarity with billing disputes cannot establish whether this particular customer was charged twice. The assistant may have learned how duplicate charges are usually discussed while having no access to the customer’s transaction record. Fluent familiarity with the category can conceal ignorance of the case.

Attention and the meaning of prediction

Attention is a computational mechanism that allows a model to combine information from different positions in its input. Learned relationships determine which information contributes to a representation at a particular stage. Layers of these operations help the system connect distant parts of a passage. In an autoregressive model generating text, the computation cannot inspect future tokens that have not yet been generated. Vaswani and colleagues’ 2017 Transformer paper made attention the central mechanism of an architecture that performed strongly on translation tasks. The technical term does not establish felt attention or conscious focus.⁴

In our example, a competent answer must keep the phrase about two charges connected to the purchase while retaining the separate urgency of rent. It must also notice any later correction. If the customer adds that one charge is marked pending, that small qualification may change the appropriate action. Producing a compassionate paragraph while overlooking it would be a substantive failure.

The phrase next-token prediction identifies a training objective and a generation procedure. On its own, it cannot settle every question about understanding. Brown and colleagues’ GPT-3 study showed that a language model could perform varied tasks when given instructions or examples in its input, without updating its weights for each task. Performance varied substantially across tasks. The study provides evidence of useful capabilities learned through language modeling, rather than a general test for subjective experience.⁵

For practical purposes, understanding can be investigated through consequences: does the system distinguish a customer’s allegation from a verified fact, handle a changed circumstance, and recognize when evidence is missing? These questions allow stronger judgments than either fluency alone or a dismissive description of the training objective. The further question of whether any of this is accompanied by experience remains separate.

This distinction keeps Walt’s situation clear. He already knows he is an agent. The revelation concerns the reader’s interpretation and Walt’s growing knowledge of how the institution has constituted his life. The novel grants access to an inner account; an actual model’s account of itself requires investigation alongside its behavior and construction.