Multicenter accuracy and interobserver agreement of spot sign identification in acute intracerebral hemorrhage
Thien J Huynh1, Matthew L Flaherty, David J Gladstone
1From the Divisions of Neuroradiology (T.J.H., T.L.C., S.P.S., R.I.A.) and Division of Neurology, Department of Medicine, and Brain Sciences Program (D.J.G.), and Department of Emergency Medicine (J.C.), Sunnybrook Health Sciences and University of Toronto, Toronto, Canada; Department of Neurology, University of Cincinnati, OH (M.L.F., J.P.B.); Departments of Clinical Neurosciences and Radiology, Hotchkiss Brain Institute, University of Calgary, Calagary, Canada (A.M.D.); Department of Neurology, University of Ottawa and Ottawa Hospital Research Institute, Ottawa, Canada (D.D.); Departments of Medicine and Neurology (A.M., S.M.D.) and Neurointerventional Radiology (P.J.M.), Royal Melbourne Hospital, University of Melbourne, Melbourne, Australia; and Department of Public Health Sciences, University of Toronto, Toronto, Canada (G.A.T.).
Background And Purpose:
Rapid, accurate, and reliable identification of the computed tomography angiography spot sign is required to identify patients with intracerebral hemorrhage for trials of acute hemostatic therapy. We sought to assess the accuracy and interobserver agreement for spot sign identification.
Methods:
A total of 131 neurology, emergency medicine, and neuroradiology staff and fellows underwent imaging certification for spot sign identification before enrolling patients in 3 trials targeting spot-positive intracerebral hemorrhage for hemostatic intervention (STOP-IT, SPOTLIGHT, STOP-AUST). Ten intracerebral hemorrhage cases (spot-positive/negative ratio, 1:1) were presented for evaluation of spot sign presence, number, and mimics. True spot positivity was determined by consensus of 2 experienced neuroradiologists. Diagnostic performance, agreement, and differences by training level were analyzed.
Results:
Mean accuracy, sensitivity, and specificity for spot sign identification were 87%, 78%, and 96%, respectively. Overall sensitivity was lower than specificity (P<0.001) because of true spot signs incorrectly perceived as spot mimics. Interobserver agreement for spot sign presence was moderate (k=0.60). When true spots were correctly identified, 81% correctly identified the presence of single or multiple spots. Median time needed to evaluate the presence of a spot sign was 1.9 minutes (interquartile range, 1.2-3.1 minutes). Diagnostic performance, interobserver agreement, and time needed for spot sign evaluation were similar among staff physicians and fellows.
Conclusions:
Accuracy for spot identification is high with opportunity for improvement in spot interpretation sensitivity and interobserver agreement particularly through greater reliance on computed tomography angiography source data and awareness of limitations of multiplanar images. Further prospective study is needed.
