Related Experiment Video
Updated: Jun 23, 2025
Isokinetic Robotic Device to Improve Test-Retest and Inter-Rater Reliability for Stretch Reflex Measurements in Stroke Patients with Spasticity
Published on: June 12, 2019
Inter-Rater and Intra-Rater Agreement in Scoring Severity of Rodent Cardiomyopathy and Relation to Artificial
Thomas J Steinbach1, Debra A Tokarz1, Caroll A Co2
1Experimental Pathology Laboratories, Inc., Research Triangle Park, North Carolina, USA.
Abstract:
We previously developed a computer-assisted image analysis algorithm to detect and quantify the microscopic features of rodent progressive cardiomyopathy (PCM) in rat heart histologic sections and validated the results with a panel of five veterinary toxicologic pathologists using a multinomial logistic model. In this study, we assessed both the inter-rater and intra-rater agreement of the pathologists and compared pathologists' ratings to the artificial intelligence (AI)-predicted scores. Pathologists and the AI algorithm were presented with 500 slides of rodent heart. They quantified the amount of cardiomyopathy in each slide. A total of 200 of these slides were novel to this study, whereas 100 slides were intentionally selected for repetition from the previous study. After a washout period of more than six months, the repeated slides were examined to assess intra-rater agreement among pathologists. We found the intra-rater agreement to be substantial, with weighted Cohen's kappa values ranging from k = 0.64 to 0.80. Intra-rater variability is not a concern for the deterministic AI. The inter-rater agreement across pathologists was moderate (Cohen's kappa k = 0.56). These results demonstrate the utility of AI algorithms as a tool for pathologists to increase sensitivity and specificity for the histopathologic assessment of the heart in toxicology studies.
Insights
An artificial intelligence (AI) algorithm aids pathologists in assessing rodent progressive cardiomyopathy (PCM). The AI demonstrated reliable performance, enhancing diagnostic accuracy in toxicologic studies.
Area of Science:
- Cardiovascular Pathology
- Toxicologic Pathology
- Computational Pathology
Background:
- Computer-assisted image analysis algorithms can detect and quantify microscopic features of rodent progressive cardiomyopathy (PCM).
- Previous validation involved a panel of veterinary toxicologic pathologists using a multinomial logistic model.
Purpose of the Study:
- To assess inter-rater and intra-rater agreement among pathologists evaluating rodent heart histology.
- To compare AI-predicted scores with pathologist ratings for PCM quantification.
- To evaluate the utility of AI as a tool to improve histopathologic assessment in toxicology.
Main Methods:
- An AI algorithm and a panel of five pathologists evaluated 500 rodent heart histologic slides for PCM.
- Inter-rater and intra-rater agreement were assessed using weighted Cohen's kappa.
- A subset of 100 slides was re-evaluated after six months to determine intra-rater reliability.
Main Results:
- Pathologist intra-rater agreement was substantial (kappa k = 0.64–0.80).
- Inter-rater agreement among pathologists was moderate (kappa k = 0.56).
- The AI algorithm provided deterministic and consistent results, unaffected by intra-rater variability.
Conclusions:
- AI algorithms can serve as valuable tools for pathologists in toxicologic studies.
- AI can enhance both sensitivity and specificity in the histopathologic assessment of cardiac conditions like PCM.
- AI-assisted evaluation shows promise for improving the consistency and accuracy of cardiac pathology assessments.
More Related Videos
04:54Author Spotlight: IntelliSleepScorer — A High-Accuracy, Accessible GUI Software for Automated Sleep Stage Scoring in Mice and its Application in Psychiatric Research
Published on: November 8, 2024
06:11Nest Building Behavior as an Early Indicator of Behavioral Deficits in Mice
Published on: October 19, 2019