WEBVTT

NOTE week04-clip2-kv-cache
NOTE Narration read by an AI-generated voice.
NOTE The clip draws its own caption along the bottom of the frame. These cues
NOTE sit there too, so that no part of the animation is ever covered.

1
00:00:00.150 --> 00:00:02.837
When a decoder writes the next token,

2
00:00:02.837 --> 00:00:07.121
its query is compared with the
keys of every earlier token,

3
00:00:07.121 --> 00:00:11.455
and their values are mixed
in. Here the note so far is:

4
00:00:11.455 --> 00:00:14.017
the patient denies chest pain.

5
00:00:14.217 --> 00:00:16.879
Those keys and values do not change,

6
00:00:16.879 --> 00:00:21.995
so the model stores them instead
of recomputing them at every step.

7
00:00:21.995 --> 00:00:28.150
This store is the KV cache, and each
new token adds its own keys and values.

8
00:00:28.350 --> 00:00:33.082
The cache grows with every token,
in every layer and every head.

9
00:00:33.082 --> 00:00:38.893
For a model shaped like Llama 3, 8B,
that is half a megabyte per token,

10
00:00:38.893 --> 00:00:46.550
so a patient record of 32,768
tokens needs 16 gigabytes.

11
00:00:46.750 --> 00:00:52.903
Grouped-query attention shrinks it. 32
query heads share 8 key and value heads,

12
00:00:52.903 --> 00:00:59.750
so only 8 are stored, and the same
record needs 4 gigabytes instead of 16.

13
00:00:59.950 --> 00:01:04.537
DeepSeek-V3 goes further with
multi-head latent attention.

14
00:01:04.537 --> 00:01:10.630
It caches one compressed vector of
576 numbers per token and layer,

15
00:01:10.630 --> 00:01:17.150
instead of 40,960: about 71 times smaller.

16
00:01:17.350 --> 00:01:20.355
Sliding-window attention
is a third approach.

17
00:01:20.355 --> 00:01:22.892
In Gemma 3, the base of MedGemma,

18
00:01:22.892 --> 00:01:29.271
5 local layers look only at the last
1,024 tokens for every global layer

19
00:01:29.271 --> 00:01:31.580
that sees everything.