Continual Referring Expression Comprehension via Dual Modular Memorization.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Referring Expression Comprehension (REC) localizes image regions based on natural language descriptions.
• Current REC models require all training data upfront, limiting practical application.
• Sequential learning in REC is challenging due to potential catastrophic forgetting of prior knowledge.

Purpose of the Study:

• To introduce Continual Referring Expression Comprehension (CREC) as a more practical setting for REC.
• To propose an effective method, Dual Modular Memorization (DMM), to mitigate catastrophic forgetting in sequential REC.
• To establish new benchmarks for evaluating CREC methods.

Main Methods:

• Proposed Dual Modular Memorization (DMM) with Implicit-Memory and Explicit-Memory modules.
• Implicit-Memory constrains parameter changes for old tasks during new task learning.
• Explicit-Memory uses a buffer pool to store and rehearse representative samples from seen tasks.
• Created three CREC benchmarks by re-splitting RefCOCO, RefCOCO+, and RefCOCOg datasets.

Main Results:

• The DMM method significantly outperformed alternative approaches on the constructed CREC benchmarks.
• Experiments were conducted using two popular REC backbones, demonstrating DMM's effectiveness.
• The proposed DMM approach successfully addressed catastrophic forgetting in sequential REC tasks.

Conclusions:

• CREC is a more practical setting for real-world REC applications.
• DMM provides an effective solution for continual learning in REC, preserving knowledge across sequential tasks.
• The public release of code and benchmarks will facilitate further research in continual referring expression comprehension.