Cognitive load classification of mixed reality human computer interaction tasks based on multimodal sensor signals

Area of Science:

• Human-Computer Interaction (HCI)
• Cognitive Science
• Wearable Technology

Background:

• Evaluating cognitive load in mixed reality (MR) is crucial for human-computer interaction (HCI).
• Existing methods for assessing cognitive load in MR are limited.
• High cognitive load negatively impacts user performance and safety.

Purpose of the Study:

• To establish an MR multimodal experimental platform to induce and measure varying cognitive load levels.
• To identify effective sensor data streams and algorithms for cognitive load classification in MR.
• To design and validate an MR digital twin factory system for real-time cognitive load warnings.

Main Methods:

• Developed an MR experimental platform with three environments to manipulate cognitive load.
• Collected physiological and device data using HoloLens 2 and wearable heart rate sensors.
• Assessed cognitive load via NASA-TLX and analyzed data using an improved Transformer-CL algorithm.

Main Results:

• Operation time increased by 49% under high cognitive load.
• High cognitive load correlated with increased anxiety, frustration, and decreased performance.
• The developed MR system achieved 95.83% accuracy in classifying cognitive load.

Conclusions:

• The MR multimodal platform effectively induces and measures cognitive load.
• The Transformer-CL algorithm and sensor data are suitable for cognitive load classification.
• The MR digital twin factory system can effectively warn users of high cognitive load, enhancing safety and performance.