Assessing Inter-Annotator Agreement for Medical Image Segmentation

Area of Science:

• Medical imaging
• Artificial Intelligence
• Computer Vision

Background:

• AI-driven medical computer vision relies on accurate data annotations.
• Variability among expert annotators introduces noise, potentially degrading AI algorithm performance.

Purpose of the Study:

• To assess, illustrate, and interpret inter-annotator agreement in medical image segmentation.
• To evaluate the impact of annotator variability on AI training data quality.

Main Methods:

• Utilized common and ranking agreement heatmaps for qualitative assessment.
• Employed extended Cohen's kappa and Fleiss' kappa for quantitative reliability evaluation.
• Applied the STAPLE algorithm to generate ground truth and compute Intersection over Union (IoU), sensitivity, and specificity.

Main Results:

• Experiments on cervical colposcopy and chest X-ray datasets demonstrated consistent inter-annotator reliability assessment.
• Combining multiple metrics is crucial for avoiding assessment bias.
• The proposed methods provide a robust framework for evaluating annotator agreement.

Conclusions:

• Accurate inter-annotator agreement assessment is vital for reliable AI model development in medical imaging.
• The combination of qualitative and quantitative metrics offers a comprehensive approach to understanding annotator variability.
• Findings highlight the importance of addressing annotator consistency for high-performing medical AI.