Multimodal Contrastive Spatiotemporal Self-Organizing Neural Networks for In-Home Activity Learning of Mild Cognitive...

In-home spatiotemporal data, such as the movement trajectory data and the spatial time series data, contains potential predictive utility for detection of geriatric conditions including Mild Cognitive Impairment (MCI), frailty, and cognitive frailty. However, few have explored spatiotemporal learning models for learning and fusion of such disparate spatiotemporal data, owing to the lack of a generalized machine learning model that can jointly model these different spatiotemporal data types. This work reports a multimodal spatiotemporal machine learning model based on a class of self-organizing neural networks that can integrate different spatiotemporal data types for MCI detection. Specifically, Episodic Memory Adaptive Resonance Theory (EM-ART) and SpatioTemporal Episodic Memory (STEM) were employed to model movement trajectory and spatial time-series data of in-home room trips, respectively. For detecting behavioral changes, a contrastive layer was added to the individual models, generating latent representative features normalized for the different layouts of homes. A three-channel Fusion ART layer was stacked above the contrastive layers, integrating the latent representative features of the two models for the MCI classification task. Using a longitudinal real-world dataset collected from high-frequency passive infrared motion sensors paired with annual neuropsychological assessments over a period of two years, the EM-ART and STEM latent representative features achieved a Receiver Operating Characteristics-Area Under the Curve (ROC-AUC) of 0.6-0.7, demonstrating the efficacy of each model for MCI detection. By integrating the two models, the multimodal contrastive model achieves predictive accuracy and F1 rates of 84.2% and 66.7%, respectively, outperforming state-of-the-art models including Support Vector Machines (SVM) and Long Short-Term Memory (LSTM) based on cyclomatic complexity features in terms of specificity and accuracy, paving the way for clinical possibility to early detect geriatric conditions, particularly the MCI.