Memory like the human brain: A framework for decoding multimodal learning of brain-visual-linguistic features.

Area of Science:

• Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Decoding human visual neural representations is crucial for advancing brain-like intelligence research.
• Current methods using fMRI/EEG align neural signals with visual/linguistic features but face challenges like representation drift and incomplete modeling of common/individual representations.

Purpose of the Study:

• To propose a novel framework, MLHuB, that mimics human brain learning mechanisms to overcome limitations in decoding neural representations.
• To enhance the stability and accuracy of aligning neural signals with visual and linguistic features.

Main Methods:

• Implemented a memory unit to consolidate acquired knowledge by reading and updating learned text-image features.
• Utilized orthogonal projection to compute common and individual features between text and images.
• Employed intra-modality mutual information maximization to regularize learning and encourage exploration of unseen knowledge.
• Integrated intra- and inter-modality mutual information maximization for a consistent joint representation.

Main Results:

• MLHuB demonstrated state-of-the-art performance on three benchmark datasets.
• The framework effectively addresses representation drift, a key challenge in continuous learning.
• MLHuB successfully disentangles common semantics from modality-specific information in text-image pairs.

Conclusions:

• The proposed MLHuB framework offers a significant advancement in decoding human visual neural representations.
• MLHuB provides a more stable and comprehensive approach to brain-like intelligence research by mimicking human learning.
• Future research can build upon MLHuB to further explore neural representations and develop more sophisticated AI.