LFSRM: Few-Shot Diagram-Sentence Matching via Local-Feedback Self-Regulating Memory

Image-sentence matching that aims to understand the correspondence between vision and language, has achieved significant progress with various deep methods trained under large-scale supervision. Different from natural images taken by camera, diagrams in the textbooks contain more graphic objects, drawings, and natural objects, and the diagram-sentence matching plays an important role in textbook understanding and question answering. However, existing matching models are not suitable for the challenging task between diagrams and sentences, due to the more serious few-shot content and incomplete description problems. In this paper, we propose a novel local-feedback self-regulating memory framework (LFSRM) for diagram-sentence matching. On one hand, LFSRM includes an external memory to store the useful multi-modal information, especially uncommon ones, to overcome the few-shot content problem, where the memory is updated flexibly according to the local-feedback from visual-textual alignment scores. On the other hand, LFSRM designs an attention mechanism on local-level alignment scores and a strengthening factor impacted on sentence-to-diagram matching direction for alleviating the incomplete description problem. Extensive experiments on three datasets show that LFSRM achieves satisfactory results on conventional image-sentence matching, and outperforms SOTA methods on few-shot image/diagram-sentence matching by a large margin.