A Supervised Text-Embedded Transformer Matching Model to Detect Fall Injuries in Medicare Data

To develop and validate a supervised text-embedded transformer matching model to identify fall injuries in Medicare data, and evaluate the model's performance -- alongside a validated rule-based algorithm-- against "ground truth" from an external reference standard (self-reported fall injuries leading to medical attention).

Text embeddings of ICD-10-CM and CPT codes in Medicare claims/encounters from participants in the Strategies to Reduce Injuries and Develop Confidence in Elders (STRIDE) trial served as model inputs. Trained on annotated claims/encounters occurring within +/- one month of self-reported fall injuries leading to medical attention, the transformer model generated a continuous 0-1 probability that each claim/encounter was for a fall injury. The model was then applied to all claims/encounters in STRIDE and compared alongside the rule-based algorithm to the external reference standard.

The model achieved an area under the curve (AUC) of > 0.96 against annotated claims/encounters in 9 out of 10 holdout folds and 0.85 in the remaining fold. In the full STRIDE dataset, the model achieved a peak AUC of 0.86 (95% CI, 0.84-0.87) against the external reference standard, with results comparable to the rule-based algorithm.

Relative to rule-based approaches, which typically generate binary outcomes, the continuous event probability generated by the transformer model could support clinical endpoint adjudication, with high-probability predictions treated as events, moderate-probability predictions being adjudicated, and low-probability predictions treated as non-events.

A text-embedded transformer model identified fall injuries with comparable accuracy to a rule-based algorithm, demonstrating "proof of concept" for use in endpoint adjudication.