Study finds wide variation in chest X-ray AI performance

Commercial AI devices varied widely in lung cancer detection on chest X-rays, according to a head-to-head study published May 19 in Radiology.

Ahmed Maiter, MB BChir, of Sheffield Teaching Hospitals NHS Foundation Trust, and colleagues tested 7 AI devices on 5,235 posteroanterior chest radiographs from 5,235 adult patients at a single U.K. center. The radiographs were acquired between July 2020 and February 2021.

Confirmed lung cancer with a visible tumor was present in 1.4% of patients. Median patient age was 60 years, 53.4% were female, and 79.4% were White.

Participating manufacturers included Annalise.ai, now Harrison.ai, Gleamer, InferVision, Milvue, Oxipit, Qure.ai, and Rayscape. Device outputs were later anonymized in the analysis.

Performance varied across key metrics. Area under the receiver operating characteristic curve ranged from 0.80 to 0.94, sensitivity from 20.8% to 77.8%, specificity from 58.9% to 98.4%, and positive predictive value from 1.5% to 28.4%.

Pairwise comparisons showed significant differences in 39 of 44 classification-result comparisons. Device classification agreement was minimal, with a Fleiss κ of 0.24.

Compared with radiologist reports, 3 devices detected more tumors and 4 detected fewer tumors. Additional false-positive results ranged from 10 to 2,039, depending on the device.

False-positive failure analysis found that erroneous detection of other pathologies was the most common cause of false-positive results, with a median of 71%.

Study authors concluded that commercial AI tools for this use case do not perform equally. They said future work should examine the impact of different devices on radiologist accuracy, reporting behavior, patient outcomes, and healthcare service delivery.