WEBVTT

NOTE week05-clip1-post-training
NOTE Narration read by an AI-generated voice.
NOTE The clip draws its own caption along the bottom of the frame. These cues
NOTE sit there too, so that no part of the animation is ever covered.

1
00:00:00.150 --> 00:00:04.094
A pretrained model only
continues the text it is given.

2
00:00:04.094 --> 00:00:08.749
We gave the base Qwen3 model the
start of a discharge summary,

3
00:00:08.749 --> 00:00:13.950
and it simply kept writing a
note about a different patient.

4
00:00:14.150 --> 00:00:16.755
Now we ask it a question about the note.

5
00:00:16.755 --> 00:00:20.077
The base model does not
answer like an assistant.

6
00:00:20.077 --> 00:00:25.750
It treats the text as a quiz and
keeps inventing new questions.

7
00:00:25.950 --> 00:00:31.463
The post-trained version of the same
model answers the question directly.

8
00:00:31.463 --> 00:00:37.401
But here its answer is wrong: the note
says the patient denies chest pain.

9
00:00:37.401 --> 00:00:41.683
Clips three and four show
two ways to fix this.

10
00:00:41.883 --> 00:00:46.701
Post-training adds stages after
pretraining: supervised fine-tuning,

11
00:00:46.701 --> 00:00:48.331
then preference tuning,

12
00:00:48.331 --> 00:00:53.008
then reinforcement learning with
rewards that a program can check.

13
00:00:53.008 --> 00:00:56.350
We cover the last stage in clip four.

14
00:00:56.550 --> 00:01:01.151
Supervised fine-tuning trains
on instruction and answer pairs.

15
00:01:01.151 --> 00:01:04.134
Each pair is wrapped in a chat template,

16
00:01:04.134 --> 00:01:07.723
with special markers for
where the user's turn

17
00:01:07.723 --> 00:01:11.350
and the assistant's turn begin and end.

18
00:01:11.550 --> 00:01:15.561
The loss is the same next-token
loss as in pretraining.

19
00:01:15.561 --> 00:01:19.268
The only difference is that it
is counted on the answer tokens,

20
00:01:19.268 --> 00:01:24.217
not on the instruction, so the
model learns to write answers.

21
00:01:24.417 --> 00:01:28.696
So the architecture is the
same and the loss is the same.

22
00:01:28.696 --> 00:01:34.846
Only the training data changes, from raw
text to instruction and answer pairs,

23
00:01:34.846 --> 00:01:39.780
and that is what turns a text
generator into an assistant.