Predicting the Value of Radiology Artificial Intelligence Applications: Large-Scale Predeployment Evaluation of a Portfolio of Models
American Journal of Roentgenology, 2026
This prospective study addresses a key challenge in radiology AI: model performance reported before deployment may not reflect the value an AI tool will provide in real clinical practice.
Larson and colleagues developed a structured predeployment evaluation method combining traditional performance metrics with “augmentation” metrics, aimed at estimating how often AI could identify findings missed by radiologists.
The study assessed 13 AI models across 12 clinical tasks in a large national radiology practice, using 88,645 examinations from multiple sites. A radiologist workgroup also evaluated the potential clinical value of AI support for each task.
Results showed that radiologists generally had higher positive predictive value, while AI models generally had higher sensitivity. The predicted value of AI support aligned with post-deployment radiologist perceptions for 10 of 12 tasks.
Overall, the study proposes a practical framework to help radiology practices decide which AI tools are likely to add meaningful value before purchasing or deploying them.