LFSRM: Few-Shot Diagram-Sentence Matching via Local-Feedback Self-Regulating Memory

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Image-sentence matching is crucial for vision-language understanding.
• Textbook diagrams present unique challenges compared to natural images, including more graphic objects and incomplete descriptions.
• Existing models struggle with few-shot content and description incompleteness in diagram-sentence matching.

Purpose of the Study:

• To propose a novel framework, the local-feedback self-regulating memory framework (LFSRM), for improved diagram-sentence matching.
• To address the few-shot content problem by incorporating external memory for multi-modal information.
• To alleviate incomplete description issues using local-level attention and a strengthening factor.

Main Methods:

• Developed a local-feedback self-regulating memory framework (LFSRM).
• Implemented an external memory module updated via local-feedback for few-shot learning.
• Incorporated an attention mechanism on local-level alignment and a strengthening factor for sentence-to-diagram matching.

Main Results:

• LFSRM demonstrates satisfactory performance on conventional image-sentence matching tasks.
• LFSRM significantly outperforms state-of-the-art (SOTA) methods on few-shot image/diagram-sentence matching.
• The proposed framework effectively handles challenges posed by textbook diagrams.

Conclusions:

• LFSRM offers a robust solution for diagram-sentence matching, particularly in low-data regimes.
• The framework's memory and attention mechanisms are key to its success.
• The AI2D dataset and LFSRM code are publicly available to facilitate further research.