Multi-modal deep learning model for bipolar depression adolescents with verbal auditory hallucinations

To develop a multimodal deep learning-based classification model for adolescent bipolar depression (ABD) with verbal auditory hallucinations (AVHs).

A retrospective analysis was conducted on 47 untreated ABD patients within 30 days, between January 2024 and August 2025. Comprehensive clinical data were collected, including sex, age, age at onset, years of education, and the presence of suicidal or self-harming behaviors. Based on the PANSS P3 score and the presence of AVHs, patients were divided into a hallucination group (P3 score > 3, n = 24) and a non-hallucination group (P3 score ≤ 3, n = 23). All participants underwent 1H-MRS scanning of the ventromedial prefrontal cortex (vmPFC). A multimodal deep learning model was constructed using MRS-derived features in combination with clinical parameters.

The model achieved an optimal classification accuracy of 71.43% on the fixed test set, as obtained by the second-fold model. This best-performing model demonstrated balanced and stable classification performance for both positive and negative samples, with precision, recall, and F1-score all reaching 0.75.

This study proposes a novel multimodal Transformer-based framework and evaluates its effectiveness in classifying ABD patients experiencing depressive episodes with AVHs. The results suggest that the advanced model architecture, incorporating mechanisms such as bidirectional cross-attention and an Ensemble of Experts classifier, can effectively integrate heterogeneous data and achieve a test accuracy of 71.43% on a small dataset, indicating preliminary technical feasibility.