Perceived Mental Workload Classification Using Intermediate Fusion Multimodal Deep Learning.

Area of Science:

• Neuroscience
• Cognitive Science
• Machine Learning

Background:

• Accurate detection of mental workload (MWL) is crucial for various applications.
• Existing research often relies on single bio-signal modalities for MWL classification.
• Deep learning offers novel approaches for analyzing complex bio-signal data.

Purpose of the Study:

• To classify perceived mental workload (PMWL) using a flexible deep neural network (DNN) that integrates multiple bio-signal modalities.
• To enable feature sharing across different data streams for enhanced classification accuracy.
• To develop a modular and generalizable DNN architecture for multimodal analysis.

Main Methods:

• An experiment was conducted using verbal logic puzzles with five difficulty levels to simulate MWL.
• Participants rated perceived difficulty on a 7-point scale.
• Four modalities were collected: galvanic skin response, photoplethysmograms, functional near-infrared spectrograms, and eye movements, alongside puzzle markers.
• A novel intermediate fusion multimodal DNN was designed and evaluated.

Main Results:

• The multimodal DNN achieved high accuracy (0.985) in classifying PMWL on a seven-level scale.
• The model demonstrated superior performance when utilizing multiple modalities compared to single modalities.
• The modular design facilitated easy addition or removal of modalities without significant architectural changes.

Conclusions:

• Multimodal bio-signal integration via a flexible DNN significantly enhances the accuracy of perceived mental workload classification.
• The developed DNN architecture is robust, modular, and generalizable for future research.
• Openly available dataset and code promote reproducibility and further advancements in the field.