Adaptive multimodal learning for driver cognitive state monitoring using transformer-based fusion with personalized...

Road accidents caused by driver fatigue and cognitive overload remain a significant public safety concern. According to recent traffic safety data, drowsy driving contributes to thousands of fatal accidents each year, emphasizing the urgent need for intelligent driver monitoring systems. To address this, we propose an adaptive multimodal deep learning framework (AML) for real-time cognitive workload assessment and fatigue detection, leveraging the CL-Drive dataset: a multimodal repository of EEG (cognitive load), ECG (cardiac activity), EDA (electrodermal arousal), and gaze tracking (visual attention) captured from 21 participants during simulated driving across nine scenarios of escalating complexity. Our framework integrates a hybrid CNN-BiLSTM architecture to extract spatiotemporal features from raw physiological signals and gaze sequences, capturing localized spatial patterns and long-term temporal dynamics. These features are fused using a transformer-based network with cross-modal attention, which models interactions between modalities (e.g., correlating gaze fixation losses with EEG theta-band surges during distraction) and yields a 3.6 percentage-point absolute accuracy improvement over the strongest conventional fusion baseline under identical evaluation. To address individual variability and privacy, we combine personalized meta-learning-adapting to new drivers with as few as five windowed samples (∼10 s of synchronized multimodal data) via episodic fine-tuning-with federated optimization, enabling decentralized model updates and reducing per-client data transfer by 38% through adaptive gradient compression. Experiments on CL-Drive demonstrate state-of-the-art performance under strictly cross-subject evaluation. Under subject-independent 5-fold cross-validation, AML achieves [Formula: see text] accuracy on binary cognitive load classification without personalization, rising to [Formula: see text] with [Formula: see text] calibration samples (∼40 s). Under the more rigorous leave-one-subject-out (LOSO) protocol, AML reaches [Formula: see text] without personalization and [Formula: see text] with [Formula: see text] personalization, an improvement of 1.6 percentage points over the strongest published LOSO baseline on this dataset with a further 6.2-11.3 percentage points gained from personalization alone across the LOSO and 5-fold protocols. The framework exhibits robustness to real-world sensor noise (e.g., EEG/EDA motion artifacts) and achieves [Formula: see text] LOSO accuracy with only [Formula: see text] samples (∼10 s of calibration per new driver), critical for scalable in-vehicle deployment. By enabling privacy-aware, real-time monitoring of driver states, this work advances intelligent vehicle safety systems and provides a blueprint for adaptive multimodal learning in human-centric AI applications.