Inter-observer Variability in the Analysis of CO-RADS Classification for COVID-19 Patients

Area of Science:

• Medical Imaging
• Radiology
• Infectious Diseases

Background:

• Computed tomography (CT) of the chest was vital for COVID-19 diagnosis early in the pandemic, especially where PCR kits were scarce.
• The COVID-19 Reporting and Data System (CO-RADS) was developed for standardized reporting of COVID-19 pneumonia on CT scans.
• High inter-observer agreement is crucial for the reliable implementation of CO-RADS.

Purpose of the Study:

• To evaluate inter-observer agreement in CO-RADS scoring of COVID-19 pneumonia on chest CT among physicians of varying specialties and experience levels.
• To assess the feasibility of applying the CO-RADS reporting system to less experienced clinicians.

Main Methods:

• Retrospective review of chest CT images from 630 RT-PCR confirmed COVID-19 patients.
• Seven observers (radiologists, house officers, pulmonologist) with diverse experience levels independently scored CT images using the CO-RADS classification.
• Analysis of inter-observer agreement across different specialty groups.

Main Results:

• Almost perfect agreement was observed among radiologists.
• Substantial agreement was found between the pulmonologist and house officers.
• Moderate-to-substantial agreement was noted across all observer groups, indicating good feasibility for CO-RADS application.
• Inter-observer variability was highest among radiologists but decreased compared to pulmonologists and house officers.

Conclusions:

• The CO-RADS system demonstrates substantial to almost perfect inter-observer agreement among physicians with varying experience levels.
• Radiologists, house officers, and pulmonologists can effectively use CO-RADS to identify typical CT features of COVID-19 lung involvement.
• CO-RADS is a feasible and reliable tool for standardized reporting of COVID-19 pneumonia on chest CT.