Inter-rater reliability for assessing intracranial collaterals in patients with acute ischemic stroke: comparing 29
Lennard Wolff1, Jiahang Su2, Derek Van Loon2
1Department of Radiology and Nuclear Medicine, Erasmus MC University Medical Center, Doctor Molewaterplein 40, 3015 GD, Rotterdam, The Netherlands. l.wolff.1@erasmusmc.nl.
Insights
Assessing collateral circulation in acute ischemic stroke patients using collateral score (CS) showed moderate inter-rater reliability. Both human raters and an automated algorithm achieved similar accuracy, especially when dichotomizing the CS.
Area of Science:
- Neurology
- Radiology
- Medical Imaging
Background:
- Collateral circulation is crucial for endovascular treatment outcomes in acute ischemic stroke.
- Accurate assessment of collateral status is vital for patient management.
Purpose of the Study:
- To evaluate the inter-rater reliability and accuracy of visual and automated collateral scoring (CS) in acute ischemic stroke patients.
- To compare the performance of radiologists and radiology residents with an automated algorithm.
Main Methods:
- Utilized CT angiography (CTA) scans from the MR CLEAN study.
- Trained radiologists and residents to assess visual collateral score (CS) on selected CTA scans.
- An automated algorithm was employed to compute CS.
- Inter-rater reliability was assessed using intraclass correlation coefficient (ICC), and accuracy was calculated.
Main Results:
- All rater groups demonstrated moderate inter-rater reliability (ICC: 0.689-0.780) compared to a reference standard.
- Overall human accuracy for CS was 65%, increasing to 88% for dichotomized CS (0-1, 2-3).
- Automated CS accuracy was 62%, reaching 90% for dichotomized CS, with no significant difference based on rater experience.
Conclusions:
- Radiologist experience does not significantly influence inter-rater reliability in collateral scoring after training.
- Automated collateral scoring demonstrates comparable accuracy to human raters, including residents and experienced radiologists.
Purpose:
Outcome of endovascular treatment in acute ischemic stroke patients is depending on the collateral circulation maintaining blood flow to the ischemic territory. We evaluated the inter-rater reliability and accuracy of raters and an automated algorithm for assessing the collateral score (CS, range: 0-3) in acute ischemic stroke patients.
Methods:
Baseline CTA scans with an intracranial anterior occlusion from the MR CLEAN study (n=500) were used. For each core lab CS, ten CTA scans with sufficient quality were randomly selected. After a training session in collateral scoring, all selected CTA scans were individually evaluated for a visual CS by three groups: 7 radiologists, 13 junior and 9 senior radiology residents. Two additional radiologists scored CS to be used as reference, with a third providing a CS to produce a 2 out of 3 consensus CS in case of disagreement. An automated algorithm was also used to compute CS. Inter-rater agreement was reported with intraclass correlation coefficient (ICC). Accuracy of visual and automated CS were calculated.
Results:
39 CTA scans were assessed (1 corrupt CTA-scan excluded). All groups showed a moderate ICC (0.689-0.780) in comparison to the reference standard. Overall human accuracy was 65± 7% and increased to 88± 5% for dichotomized CS (0-1, 2-3). Automated CS accuracy was 62%, and 90% for dichotomized CS. No significant difference in accuracy was found between groups with different levels of expertise.
Conclusion:
After training, inter-rater reliability in collateral scoring was not influenced by experience. Automated CS performs similar to residents and radiologists in determining a collateral score.
More Related Videos
08:40Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
13:50Focal Cerebral Ischemia Model by Endovascular Suture Occlusion of the Middle Cerebral Artery in the Rat
Published on: February 5, 2011