Performance and Agreement When Annotating Chest X-ray Text Reports-A Preliminary Step in the Development of a Deep...

Area of Science:

• Radiology
• Medical Informatics
• Artificial Intelligence

Background:

• Chest X-ray reports serve as crucial communication tools and data sources for developing AI-driven decision support systems.
• Consistent understanding and accurate labeling of these reports are essential for reliable AI model training.

Purpose of the Study:

• To evaluate how different annotators comprehend and label chest X-ray reports.
• To assess the performance of various medical professionals in annotating these reports for AI development.

Main Methods:

• 200 chest X-ray reports were annotated by a diverse group: board-certified radiologists, trained radiologists, radiographers, a non-radiological physician, and a medical student.
• Consensus labels from experienced radiologists defined the 'gold standard'.
• Matthew's Correlation Coefficient (MCC) and descriptive statistics were used to measure annotation performance and agreement.

Main Results:

• Intermediate radiologists achieved the highest correlation with the gold standard (MCC 0.77).
• Novice radiologists and medical students showed strong performance (MCC 0.71).
• Non-radiological annotators with general knowledge (physician, student) demonstrated better alignment with radiologists than specialized non-radiologists.

Conclusions:

• For AI development using chest X-ray reports, non-radiological annotators with general medical knowledge can provide valuable input when expert radiologists are limited.
• The findings suggest that generalist annotators may be more suitable than sub-specialized non-radiologists for certain AI training tasks in diagnostic radiology.