Multimodal Fusion of Smart Home and Text-based Behavior Markers for Clinical Assessment Prediction

Area of Science:

• Digital Health
• Machine Learning
• Behavioral Science

Background:

• Technology enables unobtrusive collection of behavioral data for various applications.
• Clinical assessments screen for impairment and disease, like mild cognitive impairment (MCI).
• Automating the mapping of behavioral data to assessment scores can aid in monitoring health and behavior changes over time.

Purpose of the Study:

• To extract behavior markers from smart home environments and a digital memory notebook app.
• To map these multimodal behavior markers to ten clinical assessments relevant for monitoring MCI onset and cognitive health.
• To utilize machine learning for fusing multimodal behavior markers and joint prediction.

Main Methods:

• Behavior markers were extracted from smart home activity patterns (hourly, daily, weekly) and app usage (usage, content, style).
• Machine learning techniques were employed to fuse these multimodal behavior markers for joint prediction.
• Three regression algorithms were used to evaluate the approach with data from 14 participants with MCI.

Main Results:

• Moderate to large correlations were observed between predicted and ground-truth assessment scores.
• Correlation coefficients ranged from r = 0.601 to r = 0.871 for each clinical assessment.
• The study demonstrated the utility of fusing multimodal behavioral data for predicting clinical assessment scores.

Conclusions:

• Unobtrusive, multimodal behavioral data can be effectively used to predict clinical assessment scores.
• This approach shows promise for early detection and continuous monitoring of mild cognitive impairment (MCI).
• Integrating smart home and digital app data offers a novel pathway for objective health assessment.