Related Experiment Video
Updated: May 31, 2026
10:28
Dynamic Digital Biomarkers of Motor and Cognitive Function in Parkinson's Disease
Published on: July 24, 2019
Humanizing "multimodal" digital phenotyping of serious mental illness
Alex S Cohen1, Michael D Masucci1, Ole E Granrud1
1Department of Psychology, Center for Computation & Technology, Louisiana State University.
Journal of Psychopathology and Clinical Science
|May 28, 2026
Summary
Digital tools can quantify serious mental illness (SMI) symptoms by integrating verbal and nonverbal cues. Nonverbal digital features linked to negative symptoms, but only when language had a positive emotional tone, offering context for human interpretation.
Area of Science:
- Digital phenotyping
- Computational psychiatry
- Mental health technology
Background:
- Quantifying serious mental illness (SMI) symptoms using digital technologies has been researched for over 70 years.
- Recent advancements focus on multimodal feature integration to capture distinct behavioral domains.
- Understanding how humans integrate verbal and nonverbal information is key for accurate clinical ratings.
Purpose of the Study:
- To evaluate the links between nonverbal digital features and human-rated negative symptoms in individuals with SMI.
- To determine if the relationship between nonverbal features and negative symptoms is moderated by the emotional tone of language.
- To approximate human integration of verbal and nonverbal information in clinical assessments.
Main Methods:
- Utilized mobile video diaries from individuals with SMI, those at clinical high risk, and control groups.
- Identified six reliable nonverbal digital features (facial expressions, vocal intonation, articulation rate, etc.).
- Behaviorally rated videos for alogia and blunted affect, analyzing relationships with digital features moderated by language tone.
Main Results:
- Most digital features lacked acceptable reliability for quantifying SMI symptoms.
- Six features demonstrated acceptable reliability, including positive/neutral facial expressions and vocal characteristics.
- Relationships between nonverbal features and negative symptoms were dependent on positive verbal tone; alogia correlated with longer pauses during positive speech.
Conclusions:
- Multimodal digital phenotyping can provide context, mirroring human interpretation of integrated information streams.
- Nonverbal digital markers of negative symptoms are influenced by the emotional valence of language.
- This approach offers a novel way to use digital tools for assessing mental illness.
