WEBVTT

NOTE week07-clip2-zero-shot
NOTE Narration read by an AI-generated voice.
NOTE The clip draws its own caption along the bottom of the frame. These cues
NOTE sit there too, so that no part of the animation is ever covered.

1
00:00:00.150 --> 00:00:06.850
Once images and text share one space,
we can classify without any training.

2
00:00:06.850 --> 00:00:11.814
We write one text prompt for
each of the eight tissue classes,

3
00:00:11.814 --> 00:00:15.683
such as an H&E image of tumor stroma.

4
00:00:15.883 --> 00:00:19.315
The text encoder turns
each prompt into a vector.

5
00:00:19.315 --> 00:00:24.696
Together, the eight vectors act as
the weights of a linear classifier,

6
00:00:24.696 --> 00:00:28.950
written in text instead
of learned from labels.

7
00:00:29.150 --> 00:00:32.348
A new tile goes through the image encoder,

8
00:00:32.348 --> 00:00:38.987
its embedding is compared with every class
vector, and a softmax picks the class.

9
00:00:38.987 --> 00:00:44.160
BiomedCLIP gives this tumor
tile 0.57 for adenocarcinoma:

10
00:00:44.160 --> 00:00:46.750
correct, but not confident.

11
00:00:46.950 --> 00:00:50.635
On natural images this
works remarkably well.

12
00:00:50.635 --> 00:00:56.876
CLIP's best model reaches 76.2% on
ImageNet without a single ImageNet label,

13
00:00:56.876 --> 00:01:02.852
matching a ResNet-50 trained
on 1.28 million labeled images.

14
00:01:02.852 --> 00:01:07.883
The same paper notes it is weak
on lymph node tumor detection.

15
00:01:08.083 --> 00:01:12.486
On our 5,000 H&E tiles in 8 classes,

16
00:01:12.486 --> 00:01:17.887
general CLIP scores between 5.6 and 15.1%,

17
00:01:17.887 --> 00:01:24.147
depending only on the wording
of the prompt. Chance is 12.5%,

18
00:01:24.147 --> 00:01:27.217
so the worst prompt is below chance.

19
00:01:27.417 --> 00:01:32.463
BiomedCLIP, trained on figures and
captions from biomedical papers,

20
00:01:32.463 --> 00:01:38.563
does much better, at 34 to 39%.
That is still far from reliable,

21
00:01:38.563 --> 00:01:43.217
and the wording of the prompt
still changes the result.

22
00:01:43.417 --> 00:01:46.809
Now train a linear probe on
the same frozen embeddings,

23
00:01:46.809 --> 00:01:49.328
with only ten labeled tiles per class.

24
00:01:49.328 --> 00:01:55.149
It reaches 64% with CLIP
and 77% with BiomedCLIP.

25
00:01:55.149 --> 00:01:58.180
A few labels beat a clever prompt.