A multimodal evidence-driven framework for clinical decision support in cognitive impairment

Deep learning approaches for cognitive impairment diagnosis have shown considerable promise, but their clinical translation remains limited by poor interpretability and weak linkage between model outputs and established medical evidence. Here we developed the Multimodal Evidence-Driven Reasoning Framework (MEDRF), which integrates a Multimodal Hierarchical Cascade (mHC) classifier with a retrieval-augmented large language model (RAG-LLM) for evidence-guided reasoning. MEDRF leverages routinely collected non-invasive data from clinical profiles and structural MRI to identify cognitive impairment stages and etiologies. Across 15 diagnostic labels, mHC outperformed flat multimodal baselines, supporting hierarchical diagnostic modeling. When the mHC was evaluated under progressive feature masking, performance declined with increasing missingness, whereas RAG-LLM correction mitigated this effect, especially under severe sparsity. In external validation on a heterogeneous cohort with primary labels, integrating RAG-LLM with the mHC improved all evaluation metrics, increasing overall accuracy from 0.706 ± 0.038 to 0.753 ± 0.032. The RAG module resolves ambiguous predictions by retrieving analogous cases and supporting textual evidence, enabling multi-hop reasoning across conflicting clinical cues. Physician review further indicated favorable quality and perceived usefulness of the generated reports. By synthesizing hierarchical prediction with evidence-grounded reasoning, MEDRF provides a robust interpretable framework for decision support, particularly in incomplete or diagnostically ambiguous presentations.