Comparison of sinus computed tomography staging systems

In an attempt to establish a standardized rating system for CT of the paranasal sinuses, the Committee on Rhinology and Paranasal Sinus Disease of the American Academy of Otolaryngology-Head and Neck Surgery instituted a protocol for the review of sinus CT scans at six international sites. Fifty identical scans were rated by four otolaryngologists at each site according to five established sinus CT staging systems. Twenty of 24 reviewers repeated the rating session at least 1 week later to determine intrarater variability. The number of CT scans that could not be classified by a particular rating system ranged from 1.3% to 5.5%. The range of intrarater agreement (kappa = 0.39 to 0.74) exceeded that of interrater agreement (kappa = 0.18 to 0.49). A skewed distribution of CT scans resulted in a system with high rater agreement but poor ability to differentiate among disease states. The use of a numeric rating system to assign a score to each scan produced a comprehensive and disease-sensitive system, but one with low rater agreement. A precise definition of mucosal thickening in terms of millimeters appeared to enhance the raters' ability to assign stage and improve a system's comprehensiveness and reproducibility. On the basis of these findings, recommendations are made for the use of CT rating systems to study clinical outcomes in patients with chronic sinusitis.