The stability paradox: Why high prediction accuracy does not guarantee reliable feature importance in psychiatric...

This study examines the critical disconnect between prediction accuracy and feature importance reliability in machine learning applications for psychiatric research. Using a hikikomori dataset with 611 instances, we compared feature selection stability across supervised models (random forest, XGBoost, logistic regression), unsupervised methods (feature agglomeration, highly variable gene selection), and statistical approaches (Spearman correlation). Despite achieving highest classification accuracy (66.20 %), logistic regression exhibited significant instability in feature rankings when the top feature was removed. In contrast, unsupervised methods and statistical approaches demonstrated perfect stability in feature ranking orders. Our findings reveal that supervised models suffer from label-driven instability while unsupervised methods provide more consistent feature importance assessments, suggesting that psychiatric researchers should supplement high-accuracy supervised models with unsupervised approaches for reliable feature interpretation.