Comparison and validation of automated scoring of SPECT phantom cold rod visibility

Evaluation of phantom image quality is an integral component of the quality assurance of SPECT systems. This evaluation often is done by visual assessment of the resolution of known structures of a specified size, such as arrays of cold rods in a warm background. Although this method is rapid and convenient, it is qualitative and is subject to inter- and intraobserver variability. Thus an automated quantitative analysis would be preferable. Several metrics of cold rod visibility have been developed, although their suitability for SPECT quality assurance depends on how well they correspond to visual scoring by experienced observers.

Various metrics of cold rod visibility, derived from either texture analysis or template-based analysis, were investigated. The texture analysis methods measured the normalized gray-level co-occurrence matrix (GLCM) energy ("Energy%") and entropy ("Entropy%") of each region and an associated combination of the two ("EnergyEntropy%"). One template-based method measured the rods-to-background contrast ("Contrast") and an associated visibility index (Contrast × area = "Contrast Visibility"). Another template-based method performed binary classification (BC) of the rods and background to compute the area under curve (AUC) of its receiver operating characteristics (ROC) curve ("BC-AUC") and the corresponding signal-to-noise ratio ("BC-SNR"). All these metrics were computed for 90 SPECT acquisitions of the standard American College of Radiology ("Jaszczak") phantom. Cold rod visibility was scored independently by two experienced nuclear medicine physicists on both dichotomous and 5-point scales. Scoring was performed twice by each observer to evaluate variability.

Interobserver agreement (Cohen's kappa statistic) was 0.78, and intraobserver reproducibility was 0.86 and 0.88, respectively, for each observer. Mean and median scores differed significantly between observers. Accuracy of each metric was assessed according to AUC of ROC analysis with respect to mean dichotomous score. The binary classification metrics had the highest accuracy (BC-AUC = 0.995, BC-SNR = 0.994), above that of the texture analysis metrics (Entropy% = 0.992, Energy% = 0.988, EnergyEntropy% = 0.992) and conventional template analysis (Contrast = 0.984, Contrast Visibility = 0.989). The metrics were similar in terms of rank correlation to mean visibility score. BC-AUC correlated linearly with mean visibility score (R²  = 0.95) and consistently performed among the highest of the metrics vs rod diameter and count level.

Automated quantitative analysis of SPECT phantom cold rods correlated well with visual scoring. The metrics based on binary classification performed particularly well for this task, across the range of rod diameters and count levels. The suboptimal interobserver agreement highlights the importance of developing automated algorithms for evaluating scanner performance.