Effect of multimodal HMI on mental workload and driving safety in smart cockpits

Multimodal human-machine interaction (HMI) is increasingly implemented in smart cockpit systems, yet empirical evidence on how specific modality configurations jointly shape driver workload and performance remains limited. From a neuroergonomic perspective, this study examined how variations in combinations of visual layout, auditory feedback, and haptic interaction were associated with driver workload regulation and vehicle-control performance in a controlled driving simulation. Twenty-nine licensed drivers performed secondary interaction tasks under systematically varied multimodal configurations, while neural activity, eye movements, physiological responses, driving performance, and subjective workload were concurrently recorded. The results indicate that different multimodal configurations do not produce uniform patterns of workload and performance outcomes; instead, different configurations elicit distinct patterns across neural, physiological, and behavioral indicators, suggesting differences in how cognitive demands may be distributed across perceptual, motor, and executive processes. Configurations characterized by more stable visual attention and direct interaction were associated with different patterns of processing demands and vehicle-control responses. Cross-indicator analysis further showed that relationships among measurement domains were selective rather than uniformly convergent. Notably, subjective workload did not always align with objective indicators, suggesting that different measures may capture partially distinct aspects of driver state. Overall, the findings highlight that the effectiveness of multimodal HMI depends on how modalities are configured rather than simply on the presence of multiple modalities, and underscore the value of interpreting driver workload through complementary rather than interchangeable indicators.