Visual and AI-Based Assessment of COVID-19 Pneumonia: Practicability and Reproducibility of an Established...

Background/Objectives: To determine the inter-rater agreement of visual and AI-based assessments of a renowned semi-quantitative chest CT scoring system (Pan-score) used to evaluate the severity of pulmonary involvement (e.g., ground-glass opacities, consolidations) in patients suffering from COVID-19. Methods: This retrospective study includes patients with PCR-confirmed COVID-19, who received a chest CT scan (not more than three days prior to or after the positive PCR test) between 21 March 2020 and 30 December 2022. The five lung lobes were scored separately on a scale from 0 (no pulmonary involvement) to 5 (>75% pulmonary involvement) by a radiology specialist, an experienced assistant physician, a medical student, and a dedicated AI-based software tool for chest CT. Weighted Cohen's κ values were calculated to assess the reliability of agreement between the different readers. Results: A total of 569 consecutive patients (381 males [67.0%], 188 females [33.0%]; mean age 68.8 years) with confirmed COVID-19 were evaluated. All of them received at least one chest CT scan. There was a significant difference (p < 0.001) between the mean Pan-score evaluated by the three human readers (9.35 ± 6.03) and the score computed fully automatically by the software (10.44 ± 5.10). However, the inter-rater agreement both between the three different human readers and between the human readers and the AI was high throughout, with κ values of 0.71-0.86 and 0.83, respectively. The slice thickness of the reconstructed CT images did not have an impact on the inter-rater agreement, but the total score was significantly higher when the images were acquired following the administration of i. v. contrast media. Conclusions: The evaluated chest CT scoring system is user-friendly due to its simplicity, though it is generally prone to inaccuracies, since the estimation of the extent of pulmonary involvement is quite subjective. Nevertheless, the inter-rater agreement was high throughout, both between the differently experienced human readers and between the human readers and the AI software. In summary, the Pan-score seems to be a reliable approach to estimate the extent of pulmonary involvement in patients suffering from COVID-19.