Panic Prediction from Digital Phenotyping: Subject-Level Cross-Validation Reveals Limited Between-Person...

Area of Science:

• Digital phenotyping
• Machine learning in healthcare
• Mental health prediction

Background:

• Digital phenotyping datasets are used to train models for predicting panic events.
• Previous models reported high accuracy (ROC-AUC ~0.90) using cross-validation.
• Concerns exist regarding potential information leakage and lack of between-person generalization in prior analyses.

Purpose of the Study:

• To reanalyze a public digital phenotyping dataset to quantify between-person generalization.
• To compare stratified cross-validation with leave-one-subject-out (LOSO) cross-validation.
• To assess the true predictive performance of models for day-before-panic events.

Main Methods:

• Reanalysis of a public dataset (3,969 person-days, 254 panic events) using an XGBoost classifier and 71 features.
• Comparison of stratified 5-fold cross-validation with LOSO cross-validation.
• Application of retrospective diagnostic decomposition with within-subject z-scoring for dynamic features.

Main Results:

• Original ROC-AUC of 0.895 ± 0.011 was reproduced.
• LOSO cross-validation resulted in a significant drop in pooled ROC-AUC to 0.489.
• SHAP analysis revealed static traits dominated feature importance under stratified CV, with weak residual within-person signal detected.

Conclusions:

• Robust between-person generalization for predicting panic events was not confirmed.
• Feature importance was dominated by static traits, not necessarily reflecting true predictive power.
• Digital phenotyping studies should prioritize subject-level cross-validation and adhere to reporting guidelines like TRIPOD and PROBAST.