Learning Low-Dimensional Semantics for Music and Language via Multi-Subject fMRI

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Linguistics

Background:

• Embodied Cognition (EC) posits that brain semantics derive from multimodal experiences.
• Individual brain biases can introduce noise, complicating media semantic extraction.
• Existing methods struggle with the complexity of neural semantic encoding.

Purpose of the Study:

• To develop a method for representing media semantics using low-dimensional vector embeddings.
• To investigate the extraction of music and language semantics from brain activity.
• To validate the proposed method using functional Magnetic Resonance Imaging (fMRI) data.

Main Methods:

• Utilized Generalized Canonical Correlation Analysis (GCCA) to model multi-subject fMRI data.
• Created low-dimensional vector embeddings to capture latent semantic representations.
• Evaluated semantic richness through music genre and language topic classification tasks.

Main Results:

• Unsupervised representations outperformed high-dimensional fMRI voxel spaces in classification.
• The proposed method demonstrated improved computational efficiency.
• Joint modeling of multiple subjects enhanced semantic richness, correlating with the number of subjects.

Conclusions:

• Demonstrated the instantiation of music and language semantics within brain dynamics.
• Provided evidence supporting multimodal embodied cognition.
• Offered a novel method for extracting media semantics from multi-subject brain activity.