Computational phenotyping and predictive modeling of outcomes using multimodal objective measures in psychiatry

Clinical decision-making in psychiatry has traditionally relied on rating scales and clinical impressions documented in the electronic health record (EHR). Yet, clinical interviews contain rich behavioral signals that remain underutilized in psychiatric care. Recent advances in artificial intelligence (AI) now enable quantification of these signals, and prior work demonstrates that computational measures of speech, language, and facial expression can inform diagnosis and estimate symptom severity. Despite this progress, most prediction efforts remain confined to single modalities and individual diagnoses and focus on diagnostic classification rather than clinically actionable outcomes such as treatment discontinuation or the need for crisis care. Here, we contextualize advances in behavioral quantification and multimodal data fusion, and present the Phenotypes REimagined to Define Clinical Treatment and Outcome Research (PREDiCTOR) study, a prospective cohort study of 2100 patients entering outpatient mental health care. PREDiCTOR is designed to develop and validate dynamic, multimodal prediction signatures that predict treatment discontinuation, emergency department visits, and hospitalizations over a one-year follow-up period. The study audiovisual recordings of clinical encounters, EHR data, cognitive assessments, smartphone passive sensing, therapeutic alliance measures, and audio/text diaries within a Contextual Bandit framework that continuously updates individualized outcome estimates as new data become available. Both interpretable features and learned embeddings are leveraged, with large language models serving as feature extractors rather than clinical decision-makers. We describe the study design, data collection, and processing pipelines, hybrid predictive modeling approach, and prospective validation strategy, and discuss the potential for translating dynamic behavioral quantification into individualized clinical prognostics.