Images as tokens: Med-PaLM M

Week 7, clip 3 of 3. This is the narration of the clip, which an AI-generated voice reads in the narrated version; the silent version shows the same steps with on-screen captions only.

CLIP can score how well each caption fits a chest X-ray, but it can only choose among captions we give it. A radiology report has to be written, word by word.

So we feed the image into a language model. A vision transformer cuts the X-ray into patches and turns each one into a vector. A projection then resizes these vectors to the size of the model's word tokens.

The image tokens go in front of the instruction, write the findings for this X-ray, and the whole sequence enters the same decoder we saw in weeks four and five. It writes the report one token at a time.

Training uses the same next-token loss as supervised fine-tuning in week five. It is counted only on the report tokens, not on the image or the instruction.

Med-PaLM M starts from PaLM-E, a Palm language model joined to a vision transformer, in sizes from 12 billion to 562 billion parameters. It is fine-tuned on 14 tasks across 7 modalities, with one set of weights for all of them.

On 246 chest X-rays, clinicians preferred Med-PaLM M's report to the radiologist's in up to 40.5% of cases. Without any training examples, it detected tuberculosis with 87.7% accuracy, against 92.6 for a specialist model.

But this was a side-by-side ranking of past cases, not use in the clinic, and in most cases the model's report was not the one preferred. We return to evaluation and deployment in week 12.