AI or Not reports 95% accuracy in deepfake X-ray benchmark

AI or Not has reported 95% overall accuracy in a benchmark evaluating its ability to detect AI-generated radiographs, according to the company. 

The San Francisco-based AI media detection firm said its platform detected 100% of synthetic X-rays in a blinded test using a curated dataset from the Radiology study “The Rise of Deepfake Medical Imaging.” 

The company said its software correctly classified 92.21% of authentic X-rays as real, corresponding to a 7.8% false-positive rate. AI or Not said the dataset and source images were not part of its model training data. 

The Radiology study, published in March 2026 by researchers at the Icahn School of Medicine at Mount Sinai, evaluated whether radiologists and multimodal large-language models could distinguish authentic radiographs from AI-generated X-rays. 

The study included 17 radiologists from 12 centers in 6 countries. Half of the 264 X-rays were authentic, and half were AI-generated, according to RSNA. 

When radiologists were not told the study’s purpose, 41% spontaneously identified AI-generated images. After being told synthetic images were present, mean accuracy rose to 75%, RSNA said. 

The accuracy of 4 multimodal large-language models, GPT-4o, GPT-5, Gemini 2.5 Pro, and Llama 4 Maverick, ranged from 57% to 85% for detecting ChatGPT-generated radiographs, according to the study summary. 

AI or Not said its benchmark was performed under blinded conditions, with image labels used only after testing to score the results. The company said the results indicate a possible role for purpose-built detection tools in medical image integrity workflows. 

The Radiology study authors warned that deepfake medical images could create risks for fraudulent litigation, cybersecurity, and the reliability of digital medical records. They suggested safeguards such as invisible watermarking and cryptographic signatures at image capture. 

The study authors also published a curated deepfake X-ray dataset with interactive quizzes for educational use.