Performance of Machine Learning Suicide Risk Models in an American Indian Population

Area of Science:

• Public Health
• Health Services Research
• Machine Learning in Healthcare

Background:

• American Indian and Alaska Native populations experience significant suicide-related inequities.
• Existing suicide risk identification tools are scarce for these specific populations.
• Machine learning models offer potential for improved suicide risk prediction.

Purpose of the Study:

• To evaluate the accuracy of established machine learning models for suicide risk identification.
• To assess model performance within a majority American Indian patient population.
• To compare machine learning models against a combined augmented screening indicator.

Main Methods:

• Secondary analysis of electronic health record data (2017-2021) from a Southwest Indian Health Service unit.
• Application and comparison of Mental Health Research Network (MHRN) and Vanderbilt University (VU) models.
• Area Under the Receiver Operating Characteristic Curve (AUROC) used to compare model performance against a 90-day suicide attempt/death outcome.

Main Results:

• The MHRN model achieved an AUROC of 0.81, outperforming the VU model (AUROC 0.68) and augmented screening (AUROC 0.66).
• Initial model calibration was poor but improved after recalibration.
• The study population was predominantly American Indian (84.7%), with a 1.9% rate of suicide attempts and 0.2% suicide deaths within 90 days.

Conclusions:

• Existing machine learning models for suicide risk identification demonstrate potential when applied to American Indian and Alaska Native populations.
• These models performed better than a combined indicator of screening results, past attempts, and recent ideation.
• Recalibration is crucial for improving model accuracy and clinical utility in this demographic.