How Language Models Work

Lesson 11

Attention

Before every guess, it works out which other words matter.

Every word gets to look at every other word and give it a score: how much does this one matter right now? That scoring is called attention. Below, one tiny word changes meaning completely, without changing at all.

A spider folded from magenta paper, eight angular legs braced.
One word, and the threads running out to every other word it is leaning on. The heavier the thread, the more that word is changing this one.

1Pick a confusing sentence

Do thisRead the sentence, then switch to the other ending underneath it. The bold word never changes. Watch which words it leans on change completely.

2The sentence (the bold word is the one we’re following; the boxed words are the ones it’s looking at)

3Now change the ending (and change nothing else)

4How much the bold word looks at each one