A benchmark score is not a clinical effect

Week 12, clip 1 of 3. This is the narration of the clip, which an AI-generated voice reads in the narrated version; the silent version shows the same steps with on-screen captions only.

A benchmark asks whether the model can answer. A trial asks whether clinicians do better with it. These are different questions, and only the second one decides whether a tool helps patients.

Goh and colleagues randomized 50 physicians: 25 to GPT-4 plus their usual resources, and 25 to usual resources alone. Both groups worked up to 6 vignettes in 60 minutes, graded blind on a structured reflection rubric.

The median diagnostic reasoning score per case was 76% with GPT-4 and 74% without. The adjusted difference was 2%age points, with a 95% interval from minus 4 to 8 and a P value of 0.60. The interval includes zero.

Nor was it faster. Physicians with the model took 519 seconds per case against 565, a difference of minus 82 seconds, with a 95% interval from minus 195 to 31. That interval includes zero too.

In an exploratory arm of 3 runs, the model alone scored a median of 92%, 16 points above the control group. But the physicians who had that same model scored 76. The gap is about how people use the tool, not about the tool.

Benchmark, retrospective study, reader study, prospective trial, then deployment with monitoring. Each rung answers a question the one below it cannot. This trial is the fourth rung, and most published results stop at the first.