What a Transformer actually does, head by head

01 · The task

A rule the model cannot memorise

To observe a mechanism you need a task whose solution you know. This one fits in a sentence: draw a random sequence of letters, of a length itself drawn at random between four and twelve, and repeat it to the end of the sequence. The first occurrence is unpredictable; everything after it follows from the context, provided you find where the current letter already appeared and copy what came next.

My first version was wrongI had fixed the motif length at sixteen. A single-layer model then reached 100%: all it had to do was look sixteen positions back, a plain offset. The negative control controlled nothing. Drawing the period at random for every example removes that shortcut.

Sequence

Fig. 1 — The model runs in this page. Each cell shows the letter and, below it, what the model predicts next with its probability; tap one to see where each head looks. The “ablate” boxes switch a head off: ablate head 1 of layer 0 and watch copying collapse.

02 · The emergence

The ability appears all at once, and not with one layer

Training takes 2.3 minutes on a laptop with no graphics card. Nothing happens for two hundred steps, then copying appears abruptly and saturates at 99.7%. The same model deprived of one layer tops out at 40.4%: in-context copying needs two stages, one to record where you come from, the other to fetch what follows — the circuit described by Elhage and colleagues in 2021, then by Olsson in 2022, reproduced here at toy scale.

Accuracy on copyable positions during training, two layers against one — **Fig. 2** — Accuracy on positions whose continuation is readable from the context. The dashed line is chance (1/26).

03 · The ablation

Ablate one head, and the model can no longer copy

The culprits can be named. Each head is switched off in turn — its output set to zero — and we measure what the model loses. We also measure where its attention goes: on the previous token, or on the positions that follow earlier occurrences of the current letter.

Ablated head	Copying left	Att. previous token	Att. induction
layer 0, head 1	0,247	0,91	0,04
layer 0, head 0	0,921	0,03	0,12
layer 1, head 0	0,955	0,00	0,69
layer 1, head 1	0,981	0,00	0,70
layer 1, both heads	0,055	—	—

The reading is unambiguous. Head 1 of layer 0 places 91% of its attention on the previous token: it is the one writing, into every position, the identity of the one before it. Ablating it drops copying from 99.5% to 24.7%. The two heads of layer 1 put nearly 70% of their attention on the induction pattern — but ablating them one at a time costs almost nothing (−4 and −1.4 points): they duplicate each other. Ablated together, copying falls to 5.5%. One function, two copies, and a preparatory stage without which nothing works.

04 · A detail that is not one

The model bets on the motif coming back

On the so-called unpredictable positions, the model scores 14%, well above chance (3.8%). Looking closer, those 14% come from one place only: the motif boundary, the position where the repetition starts. There, the model is right 93.9% of the time; everywhere else it is exactly at chance (3.8%).

So it learned a betting strategy: as long as no repetition has been detected, propose the first letter of the sequence. Wrong almost everywhere, and right precisely where it counts. A number that looked like noise was in fact describing a behaviour.

Finally, the model running in this page is the one that was trained: over six sequences and 192 positions, the largest gap between the JavaScript port and PyTorch is 10⁻⁵ on the logits and on the attention maps, ablation included, and no prediction differs.