Misleading Results in Posttraumatic Stress Disorder Predictive Models Using Electronic Health Record Data: Algorithm...

Area of Science:

• Computational psychiatry
• Health informatics
• Machine learning in healthcare

Background:

• Electronic health records (EHR) are increasingly used for predictive modeling of posttraumatic stress disorder (PTSD).
• EHR data may contain biases, including diagnostic misclassification and differential healthcare utilization.
• The accuracy of EHR-based PTSD prediction versus rigorous diagnostic criteria is not well understood.

Purpose of the Study:

• Compare predictive model performance for EHR-based vs. semistructured interview-based PTSD diagnosis.
• Quantify discrepancies in model performance.
• Investigate mechanisms driving performance differences.

Main Methods:

• Compared machine learning models (elastic net, random forest, XGBoost) predicting PTSD from EHR data versus Structured Clinical Interview for DSM-5 (SCID-5) in 1343 US veterans.
• Utilized a nested cross-validation scheme.
• Models used 29 EHR visit count variables and 8 demographic variables.

Main Results:

• EHR-based PTSD diagnosis showed 73.3% concordance with SCID-5, with 17.8% false negatives and 8.9% false positives.
• Models predicting EHR-based PTSD achieved high performance (AUC .85–.9; MCC .58–.69).
• Models predicting SCID-5-based PTSD had moderate performance (AUC .71–.76; MCC .24–.28), influenced by healthcare visit frequency.

Conclusions:

• Predictive models for PTSD using only EHR data exhibit inflated performance metrics compared to those using structured clinical interviews.
• Performance discrepancies stem from circular relationships between healthcare use and EHR-based diagnoses.
• EHR-based diagnoses may not be a sufficient proxy for true PTSD criteria; caution is advised when interpreting EHR-only predictive models.