Humanizing "multimodal" digital phenotyping of serious mental illness

Developing digital technologies for quantifying symptoms of serious mental illness (SMI) has been a focus of research for over 7 decades. Recent efforts have focused on multimodal feature integration, which captures conceptually distinct behavioral domains. We evaluated links between nonverbal digital features and human-rated negative symptoms, with the expectation that their relationship would depend on verbal information (i.e., be moderated by language emotional tone). We believe this better approximates how humans integrate verbal and nonverbal information when making clinical ratings. Mobile video diaries were evaluated for people with SMI, those at clinical high risk for developing SMI, and control groups (Ns = 48, 20, 36, and 21, respectively; K videos = 902, 440, 602, and 399, respectively). We identified six features (i.e., capturing positive and neutral facial expressions, vocal intonation and emphasis, and speaking pause length and articulation rate) that showed acceptable reliability (final N and K for video/audio analysis = 109/912 and 120/2,200, respectively), but most features failed to show acceptable reliability. Videos were behaviorally rated for alogia and blunted vocal and facial affect. Relationships between nonverbal features and human ratings were dependent on verbal tone, but only when verbal tone was positive in valence. For example, behaviorally rated alogia was associated with longer pauses, but primarily when language had a positive tone. These results support a relatively novel approach to multimodal digital phenotyping, one that emphasizes using multimodal features to provide "context" in the way that humans likely interpret and integrate multimodal streams of information. (PsycInfo Database Record (c) 2026 APA, all rights reserved).