Multimodal Cognitive Load Estimation With Radio Frequency Sensing and Pupillometry in Complex Auditory Environments

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Detecting listening effort and cognitive load (CL) is challenging with current privacy-invasive and complex methods.
• Existing techniques face issues with synchronization, data alignment, and accessibility, leading to inaccurate CL estimates.

Purpose of the Study:

• To develop a multi-modal, non-invasive, and privacy-preserving approach for estimating cognitive load (CL) and listening effort.
• To combine Radio Frequency (RF) sensing with pupillometry for a more robust CL assessment.

Main Methods:

• Designed custom RF sensors to measure blood flow changes in brain regions with high spatial resolution.
• Integrated RF sensing with pupillometry (measuring pupil size and dilation) for multi-modal fusion.
• Collected a novel multi-modal dataset in a controlled environment with varying noise levels.

Main Results:

• Pupillometry data showed high reliability (average ICC > 0.95).
• Strong correlation established between pupillometry and RF data (average PCC > 0.79).
• K-means clustering successfully classified CL into high and low categories using RF data.

Conclusions:

• The combined RF and pupillometry approach provides a robust and accurate method for estimating listening effort and cognitive load.
• This non-invasive technique overcomes limitations of conventional methods.
• Future applications include integrating RF sensors into glasses for hearing aid users and optimizing speech enhancement.