Radiology image interpretation system: modified observer performance study of an image interpretation expert system.

Area of Science:

• Medical Imaging Informatics
• Artificial Intelligence in Radiology
• Diagnostic Decision Support Systems

Background:

• Computer-based expert systems offer valuable diagnostic support in medicine, particularly in radiology.
• The Radiology Image Interpretation System (RIIS) is an expert system designed for diagnosing focal bone abnormalities.

Purpose of the Study:

• To compare the diagnostic efficacy of the Radiology Image Interpretation System (RIIS) against radiology residents.
• To evaluate the performance of RIIS in interpreting focal bone abnormalities using a modified observer-performance study.

Main Methods:

• A modified observer-performance study was conducted using a dataset of 44 abnormal and 10 normal radiographs.
• Four inexperienced and five experienced radiology residents, along with the RIIS, interpreted the cases.
• Modified receiver operating characteristic (ROC) curves were generated to estimate true-positive rates at specific false-positive rates (0.05, 0.15, 0.20) and compared using a paired t test.

Main Results:

• On average, RIIS demonstrated lower accuracy compared to both experienced and inexperienced residents.
• The difference in accuracy between RIIS and experienced residents was statistically significant only at a false-positive rate of 0.05.
• RIIS performance improved when utilized by experienced residents, but this enhancement was not statistically significant compared to inexperienced residents.

Conclusions:

• The Radiology Image Interpretation System (RIIS) shows potential but is currently less accurate than experienced radiologists for focal bone abnormality diagnosis.
• Further development and validation are needed to enhance the accuracy and clinical utility of AI-driven diagnostic systems like RIIS.
• The study highlights the importance of user experience in the performance of AI diagnostic tools in radiology.