Impact of Radiologist Experience on AI Annotation Quality in Chest Radiographs: A Comparative Analysis

Area of Science:

• Medical Imaging
• Artificial Intelligence (AI)
• Radiology

Background:

• High-quality annotations are essential for training AI models in medical imaging.
• Annotation is time-consuming, limiting the availability of large datasets.
• Radiologist experience is a key factor influencing annotation quality.

Purpose of the Study:

• To investigate the impact of radiologist experience on the quality of medical image annotations.
• To compare annotation performance across different experience levels: medical students, junior professionals, and senior professionals.

Main Methods:

• 53 anonymized chest radiographs were annotated by 15 readers with varying expertise.
• Annotations included anatomical structures, pneumonic opacities, and central venous catheters (CVC).
• Annotation quality was assessed using Dice coefficient (DSC) and Hausdorff distance (HD) against a gold standard.

Main Results:

• Senior professionals generally achieved better annotation quality, especially for complex structures.
• Medical students exhibited higher variability in their annotations.
• Experience positively correlated with annotation quality for CVCs and lung structures.

Conclusions:

• Radiologist experience significantly influences the accuracy and consistency of medical image annotations.
• A mixed-expertise approach, leveraging senior radiologists for complex tasks and less experienced ones for simpler annotations, is recommended.
• Optimizing annotation processes is crucial for efficient AI model development in radiology.