Few-Shot Learning Based on Multimodal Information Processing.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Few-shot learning (FSL) models struggle with category representativeness due to reliance on limited visual data.
• Human learning effectively uses multimodal information for robust category feature representation.

Purpose of the Study:

• To emulate human multimodal learning for enhanced FSL by integrating visual and textual information.
• To develop a model capable of acquiring more representative and robust category features from limited samples.

Main Methods:

• Introduced a novel visual-semantic fusion selection mechanism (VSFSM) comprising FS-Module and CE-Module.
• FS-Module fuses and aligns visual and semantic features, performing selection and reconstruction to generate representative features and reduce noise.
• CE-Module emphasizes category-specific features in query images, reducing noise interference and yielding representative visual-semantic features.

Main Results:

• The VSFSM effectively enhances category representativeness and robustness in few-shot learning.
• The proposed method demonstrates superior classification performance compared to existing approaches.
• Ablation studies and comparative experiments on multiple datasets validate the model's effectiveness.

Conclusions:

• Integrating visual and textual information via the VSFSM significantly improves few-shot learning capabilities.
• The proposed approach offers a promising direction for developing more generalizable and robust FSL models.
• The novel objective loss function further optimizes training for enhanced performance.