Artificial Intelligence-Empowered Multimodal Learning in Psychiatry: A Scoping Review

Psychiatric disorders are highly heterogeneous, shaped by intricate interactions among genetic, neurobiological, cognitive, social, and behavioral determinants. Traditional unimodal approaches often struggle to capture the complexity, limiting diagnostic precision and prognostic accuracy. The integration of diverse data modalities offers substantial promise for advancing precision psychiatry, enabling more nuanced, comprehensive, individualized insights into mental illnesses. Recent developments in artificial intelligence (AI), particularly machine learning (ML) and deep learning (DL), have accelerated the ability to synthesize and harness complex, multimodal datasets at scale. In this review, we systematically examine the integration and analysis of multimodal data sources through advanced AI models in psychiatry research, addressing 3 key dimensions: 1) primary data modalities, including texts (e.g., clinical assessment and questionnaires), neuroimaging, electrophysiological signals, audio and video recordings, and molecular and multi-omics profiles; 2) AI-based multimodal learning approaches, encompassing feature fusion strategies and state-of-the-art methodological paradigms ranging from conventional ML to DL and transformers; and 3) representative applications, spanning present-state characterization (diagnosis, subtyping, and stratification) to future-state prediction (risk, treatment response, and prognosis). Furthermore, we critique critical challenges that impede progress, including data-related barriers (unpaired modality, limited availability, and integration complexity) and model-related limitations (generalizability, interpretability, and clinical trustworthiness). Finally, we explore future opportunities, particularly multimodal large language models that offer unprecedented ingestion and reasoning capabilities across diverse modalities. We emphasize potential pathways (development of well-linked multimodal datasets, methodological innovation, and interdisciplinary collaboration) to realize the transformative potential of AI-empowered multimodal learning, thereby advancing personalized diagnostics, prognostics, and therapeutic strategies in precision psychiatry.