Evidence for compositionality in fMRI visual representations via Brain Algebra

Area of Science:

• Cognitive Neuroscience
• Computational Neuroscience

Background:

• Neuroimaging and electrophysiology reveal how the brain encodes visual categories.
• A key question is how the brain represents combinations of multiple visual concepts.

Purpose of the Study:

• To investigate if brain responses to individual concepts are composed algebraically.
• To explore the mechanisms of concept compositionality in neural representations.

Main Methods:

• Generated "conceptual perturbations" in neural space by averaging fMRI responses to concept-specific images.
• Applied these perturbations to base image neural patterns to create new, concept-infused patterns.
• Used a pretrained fMRI-to-image decoding model to interpret the modified brain patterns.

Main Results:

• Decoding modified brain patterns yielded images reflecting combined concepts (e.g., man on skateboard + winter concept = man on snowboard in winter).
• Compositional processes in neural representations led to predictable perceptual outcomes.
• Even small perturbations in neural patterns could significantly alter decoded concepts.

Conclusions:

• The brain's combinatory encoding of concepts may follow a systematic, algebraic-like process ("brain algebra").
• This model-driven study suggests predictable perceptual outcomes from neural compositionality.
• Opens avenues for future empirical research into brain compositionality mechanisms.