WEBVTT

NOTE week02-clip2-attention-one-token
NOTE Narration read by an AI-generated voice.
NOTE The clip draws its own caption along the bottom of the frame. These cues
NOTE sit there too, so that no part of the animation is ever covered.

1
00:00:00.150 --> 00:00:04.249
Let's follow the token
cold. Its query vector,

2
00:00:04.249 --> 00:00:10.950
in red, is compared with the key vector
of every token in the sentence, in green.

3
00:00:11.150 --> 00:00:15.179
Each score is the query
for cold dotted with a key,

4
00:00:15.179 --> 00:00:20.991
then divided by the square root of
d. The scaled scores are minus 0.5,

5
00:00:20.991 --> 00:00:27.150
0.2, 1.3, 0.6, and 1.6.

6
00:00:27.350 --> 00:00:31.854
The highest scores go to cold
itself, at 1.6, and to has,

7
00:00:31.854 --> 00:00:37.483
at 1.3. These are the tokens
that matter most here.

8
00:00:37.683 --> 00:00:41.485
A softmax turns the five
scores into attention weights.

9
00:00:41.485 --> 00:00:46.883
They are all between zero and
one, and together they sum to one.

10
00:00:47.083 --> 00:00:52.264
The new vector for cold is the
weighted sum of the five value vectors,

11
00:00:52.264 --> 00:00:57.871
in blue. The values of has
and cold contribute the most.

12
00:01:00.083 --> 00:01:04.692
Now the vector for cold carries its
context. In the phrase has a cold,

13
00:01:04.692 --> 00:01:09.480
the model can represent cold as
an infection, not a temperature.