Population-Level Digital Stroke Surveillance: Building a Fair and Accurate ICD-10 Detection Model

The International Classification of Diseases, 10th Revision (ICD-10), is widely used for clinical care, quality assurance, and stroke research. Its ubiquity across healthcare systems makes it an attractive foundation for digital health tools that can support stroke surveillance and population health monitoring. However, a major limitation is that stroke detection algorithms derived from ICD codes have been developed primarily in socially homogenous populations, raising concerns about generalizability and fairness across racially diverse populations.

We developed and validated an acute ischemic stroke (AIS) detection algorithm using Classification and Regression Tree (CART) supervised machine learning, using a diverse derivation cohort. Input variables consisted of diagnostic and procedural ICD-10 codes, stratified by position and presence on admission. The model was trained on 75% and tested on 25% of the derivation cohort and externally validated in a second tertiary institution serving patients living in predominantly underrepresented and socially vulnerable communities. Performance of the algorithm was measured by sensitivity, specificity, positive predictive value (PPV), and Cohen's κ. Subgroup analyses were conducted by sex and race/ethnicity.

In the derivation cohort, the CART model achieved sensitivity of 96%, specificity of 90%, PPV of 99%, and κ = 0.78. Applied to the independent validation cohort, the algorithm identified 1,050 AIS cases and 1,664 non-AIS cases, with sensitivity 89%, specificity 95%, PPV of 92%, and κ = 0.84. Performance was comparable between women and men (κ = 0.80 for both), and strong across Black (κ = 0.81), Hispanic (κ = 0.76), and White (κ = 0.80) subgroups. Lower accuracy was observed in the Asian subgroup (κ = 0.73, PPV = 62%).

Our findings demonstrate that CART-based algorithms can provide accurate and interpretable AIS detection using ICD-10 data while explicitly addressing social fairness. The algorithm's reproducibility across independent and diverse populations highlights its potential as a low-friction, scalable, and cost-efficient tool for clinical care, surveillance, and quality improvement. Importantly, subgroup analyses underscore the necessity of ongoing fairness evaluation as performance varied by race/ethnicity, particularly in the Asian subgroup. Limitations include potential missed cases in the gold standard, lack of confidence intervals due to retrospective data, and dependence on local coding practices. This study shows that ICD-10-based machine learning algorithms, specifically CART, can serve as a model for developing an accurate and equitable digital health platform for AIS surveillance.