Multisensory naturalistic decoding with high-density diffuse optical tomography

Area of Science:

• Neuroscience
• Biomedical Engineering
• Signal Processing

Background:

• Decoding naturalistic content from brain activity holds significant neuroscience and clinical potential.
• Existing methods like fMRI and electrocorticography are limited by logistical constraints.
• High-density diffuse optical tomography (HD-DOT) offers comparable image quality to fMRI in a more accessible format.

Purpose of the Study:

• To establish the feasibility of decoding naturalistic auditory and multisensory stimuli using HD-DOT.
• To evaluate the performance of HD-DOT in decoding complex audiovisual information from brain activity.
• To investigate the impact of parameters like trial duration and number of choices on decoding accuracy.

Main Methods:

• Utilized HD-DOT data from participants viewing movie clips.
• Employed a template-matching strategy for decoding audiovisual stimuli.
• Systematically analyzed factors influencing decoding performance, including trial duration and decoding choices.

Main Results:

• Achieved 94.2% decoding accuracy for four-way classification with 4-minute trials.
• Maintained significant above-chance decoding performance with 15-second trials and up to 32 choices.
• Obtained comparable accuracies when focusing cortical sampling on visual and auditory regions, and with unimodal stimuli.

Conclusions:

• HD-DOT data possesses sufficient resolution for decoding complex, naturalistic, multisensory stimuli.
• Template matching is a viable strategy for audiovisual decoding with HD-DOT.
• These findings support future research into advanced decoding algorithms for reconstructing complex naturalistic stimuli.