Beyond feature importance: Validating true clinical associations in AI-driven psychiatric research through...

Current AI algorithms lack robust methods to validate genuine clinical associations, which require consistency and dose-response relationships while existing studies failed due to the absence of validation. This paper introduces a leave-top1-out validation approach that systematically removes the highest-ranked feature to assess ranking stability across supervised and unsupervised models. We demonstrate theoretically why supervised feature importances are unreliable due to target-driven biases and absence of ground truth validation. Using a publicly available sleep disorder dataset (374 samples, 13 features), we empirically show XGBoost achieves high accuracy (0.9198) but exhibits poor ranking stability, while Spearman correlation maintains perfect stability with competitive accuracy (0.9090). Unsupervised models demonstrate perfect consistency despite lower accuracy (0.8797-0.8850). Our findings suggest ranking stability, rather than predictive performance alone, better identifies true clinical associations, providing a rigorous multifaced framework for causal inference in medical AI applications.