Related Experiment Video
Updated: Mar 15, 2026
The Immersive Cleveland Clinic Virtual Reality Shopping Platform for the Assessment of Instrumental Activities of Daily Living
Published on: July 28, 2022
Dual modal data-driven virtual reality-based mild cognitive Impairment assessment using MCIformer
Yanjie Zhang1, Yang Pan2, Shanshan Feng3
1Department of Aeronautical and Aviation Engineering, The Hong Kong Polytechnic University, 999077, Hong Kong Special Administrative Region.
Background:
Mild Cognitive Impairment (MCI) assessment is critical for identifying cognitive decline and enabling early intervention to reduce the risk of Alzheimer's disease (AD). Virtual Reality (VR)-based cognitive assessments offer enhanced engagement, ecological validity, and user-friendliness. However, most existing VR-based systems rely primarily on task performance metrics, overlooking subtle changes in motor behaviour and underlying neural activity patterns that are indicative of early cognitive decline.
Methods:
We developed a dual-modal, data-driven VR-based MCI assessment method integrating kinematic and functional near-infrared spectroscopy (fNIRS) data. A VR-based experiment involving healthy and MCI participants was conducted to collect synchronised kinematic and neural data. Kinematic features-smoothness, coordination, and stability-were extracted from segmented movement trajectories and structured as time series to capture subtle motor deficits. Simultaneously, multichannel fNIRS data were represented as functional brain networks to assess interregional connectivity changes associated with cognitive impairment. We proposed MCIformer, a dual-modal fusion model that applies a Transformer to kinematic sequences to capture dynamic motor patterns and a Graph Transformer to fNIRS networks to detect connectivity alterations.
Results:
Experimental results show that the proposed system achieved 90% accuracy, significantly outperforming models using only kinematic data (80%) or only fNIRS data (85%). The findings demonstrate that integrating temporal motor patterns and spatial brain connectivity patterns enhances classification performance by capturing complementary brain-behaviour information.
Conclusions:
The proposed VR-based dual-modal MCI assessment approach demonstrates strong potential for scalable, accurate early diagnosis in community settings, supporting the development of more comprehensive brain-behaviour monitoring systems for cognitive health.
