Evaluating Artificial Intelligence for Chest X-Ray Analysis in a Low-Prevalence Outpatient Settings: A Multi-reader, Multi-case Study
Abstract
Chest X-ray (CXR) is frequently used in outpatient care, where clinically relevant abnormalities are rare and the utility of artificial intelligence (AI) depends on performance in a low-prevalence case mix. We conducted a single-centre, retrospective multi-reader, multi-case (MRMC) diagnostic accuracy study of 213 consecutive posterior–anterior (PA) CXRs acquired at an outpatient clinic. Six radiologists (RAD 1–RAD 6; 1–9 years of experience) and an AI system (Carebot AI CXR, Carebot s.r.o., Czech Republic) independently assessed four most common radiological findings: atelectasis, consolidation, pleural effusion, and pulmonary lesions. Ground truth was established by two thoracic radiologists with third-reader adjudication of all disagreements. Sensitivity, specificity, PPV and NPV were reported with exact two-sided 95% confidence intervals, and paired differences between each radiologist and the AI were evaluated using McNemar tests. Adjudicated prevalence was low for all findings (3.3–7.5%). Across the four findings, the AI achieved sensitivity 0.80–1.00 and specificity 0.92–0.98, with consistently high NPV (0.985–1.000) and lower PPV for rare findings (pulmonary lesions: 0.261). Radiologists demonstrated similarly high specificity (0.90–1.00) and high NPV (0.943–0.990), but with greater variability in sensitivity (0.200–0.875) depending on the finding. Most paired comparisons did not demonstrate statistically significant differences, indicating that observed disagreements were limited in this cohort. In low-prevalence outpatient setting, the AI system’s robust specificity and very high NPV support its potential role as a second-reading decision-support tool, particularly for safe exclusion of negative findings.
Find the study here.
