Measuring performance in the interpretation of chest radiographs: a pilot study

Area of Science:

• Radiology
• Medical Imaging Analysis
• Diagnostic Performance Evaluation

Background:

• Accurate interpretation of chest radiographs is crucial for diagnosing malignancy.
• Objective assessment of radiologist performance is needed to identify areas for improvement.

Purpose of the Study:

• To develop and evaluate a system for assessing radiologist performance in identifying signs of malignancy on chest radiographs.
• To quantify differences in image interpretation skills among radiologists with varying experience levels.

Main Methods:

• A test set of 30 chest radiographs (11 normal, 19 abnormal with biopsy-proven malignancy) was used.
• Fourteen radiologists (6 consultants, 8 registrars) interpreted the images and rated suspicion levels.
• Performance metrics, including sensitivity and specificity, were automatically calculated, with instant feedback provided.

Main Results:

• Consultant radiologists demonstrated significantly better performance (AUC=0.9297) than registrars (AUC=0.7648) (p=0.003).
• A significant positive correlation was found between years of experience and performance (r=0.573, p=0.032).
• Consultants completed the test faster (19.65 min) than registrars (26.51 min) (p=0.033).

Conclusions:

• A test set can effectively measure individual differences in chest radiograph interpretation.
• This system shows potential as a valuable tool for radiologist performance testing and quality assurance.