Cross-Modal Learning from fMRI to DOT: Leveraging Large-Scale Neuroimaging for fNIRS Decoding

Functional near-infrared spectroscopy (fNIRS) and its tomographic extension, diffuse optical tomography (DOT), offer portable and non-invasive measures of functional brain hemodynamics with great potential for everyday neuroimaging. However, the present scarcity of publicly available DOT data hinders deep learning model generalization and hence limits the development of its full potential. In contrast, functional MRI (fMRI) provides high-resolution hemodynamic data at scale. Here, we propose a cross-modal learning framework that leverages large-scale fMRI data to enhance DOT-based neural decoding by mapping both modalities into a shared cortical parcellation space. Using the Schaefer 600-parcel, 17-network atlas, we integrate DOT and fMRI signals into a common representation, enabling direct data fusion and transfer learning. We evaluate (i) parcel-space data pooling, demonstrating improved robustness in low-sample DOT settings, and (ii) cross-modal pretraining, where a transformer trained on 15 classes from the Human Connectome Project (HCP) dataset transfers representational structure to DOT tasks. Across four independent DOT datasets and evaluated on unseen subjects, fine-tuning consistently improves decoding performance by up to 12%. These findings show that fMRI can serve as a scalable source of representational priors for DOT, supporting multimodal neuroimaging integration.