Effect of observer instruction on ROC study of chest images

Area of Science:

• Medical Imaging
• Radiology
• Biostatistics

Background:

• Receiver-operating characteristic (ROC) analysis is widely used in medical imaging.
• Reader training is often employed to improve confidence rating distribution and data analysis in ROC studies.
• Limited experimental data exists on the impact of reader training on ROC study outcomes.

Purpose of the Study:

• To investigate the effects of reader training on the accuracy of diagnostic performance in ROC studies.
• To assess whether specific training for confidence ratings influences radiologist accuracy across different chest imaging findings.

Main Methods:

• A multi-observer, multi-disease study involving four radiologists rating 300 chest images.
• Radiologists provided ratings before and after undergoing training for well-distributed confidence ratings.
• Statistical analysis was performed to compare accuracy metrics pre- and post-training.

Main Results:

• Reader and disease-specific accuracy remained largely unaffected for interstitial disease and pneumothoraces.
• Significant accuracy changes were observed for nodule detection in two readers (P < 0.05).
• Overall accuracy was significantly affected for one reader in normal versus abnormal image classification (P < 0.01).

Conclusions:

• Reader training for ROC analysis can have variable effects on diagnostic accuracy, impacting certain tasks more than others.
• The findings highlight the need to carefully evaluate the potential influence of interventions like reader training on ROC study results.
• Researchers should consider the implications of reader training in free-reading environments to ensure the validity of ROC analysis outcomes.