Inter- and Intra-Observer Agreement When Using a Diagnostic Labeling Scheme for Annotating Findings on Chest...

Area of Science:

• Radiology
• Medical Imaging
• Artificial Intelligence

Background:

• Consistent data annotation is crucial for developing AI-based decision support systems in radiology.
• Standardized terminology in image annotation is key to achieving reliable AI model training and testing.

Purpose of the Study:

• To evaluate annotation consistency among radiologists using a novel labeling scheme for chest X-rays.
• To compare the agreement levels achieved with descriptive versus interpretive labels.

Main Methods:

• Six radiologists with varying experience annotated 100 anonymized chest X-rays twice.
• Statistical analysis included Randolph's kappa and PABAK for inter- and intra-rater agreement.
• Proportions of specific agreements were calculated for each label.

Main Results:

• Fair-to-excellent agreement (Randolph's Kappa, 0.40-0.99) was observed across all labels.
• PABAK values ranged from 0.12 to 1 for inter-rater and 0.26 to 1 for intra-rater agreement.
• Descriptive and broad labels showed the highest agreement rates.

Conclusions:

• Descriptive labeling schemes can improve annotation consistency among radiologists of different experience levels.
• Using descriptive labels may be more effective than interpretive labels for enhancing agreement in chest X-ray annotation.
• Standardized, descriptive annotation is vital for robust AI development in diagnostic radiology.