Automated MoCA scoring for Arabic speakers using hybrid AI of multimodal speech, vision, and LLM integration

Early detection of dementia and mild cognitive impairment (MCI) remains a significant clinical challenge, particularly in Arabic and resource-limited settings where culturally adapted screening tools are scarce and access to specialized neuropsychological services is constrained. The Montreal Cognitive Assessment (MoCA) is a widely validated instrument for detecting MCI; however, its administration is largely manual, limiting scalability for large-scale community screening.

In this study, we present a preliminary feasibility study of an AI-powered hybrid multimodal cognitive screening system that digitizes the Arabic version of the MoCA and integrates speech processing, computer vision, and large language model-based reasoning within a unified platform. The system captures verbal and visuospatial responses via smartphone sensors, extracts structured linguistic and geometric features, and performs cognitive state classification using a Qwen-based structured reasoning framework with emphasis on transparency and interpretability. The system was evaluated using a custom Arabic dataset of 24 participants from an elderly care facility in Palestine, including cognitively normal individuals, those with MCI, and dementia cases. This pilot evaluation is intended to establish initial feasibility rather than definitive clinical equivalence, and the findings require validation through larger, adequately powered multisite investigations.

The proposed hybrid approach achieved an overall diagnostic agreement of 83.3% with manual clinical scoring, a Cohen's κ of 0.74 (substantial agreement), and demonstrated complete output stability across five independent runs. No severe cross-category misclassifications (i.e., from Dementia to Normal) were observed. After excluding two technically compromised audio recordings, adjusted accuracy reached 90.9% with κ = 0.86. These findings support the feasibility of developing interpretable, culturally adapted AI tools for scalable cognitive screening among Arabic-speaking populations in low-resource environments. The proposed system is intended as an assistive first-line screening tool and is not a replacement for professional medical evaluation.