VLT: Vision-Language Transformer and Query Generation for Referring Segmentation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring segmentation requires understanding complex vision-language interactions.
• Existing transformer models have limitations in handling diverse and dynamic language expressions due to fixed queries.
• A robust framework is needed to dynamically interpret language for accurate image segmentation.

Purpose of the Study:

• To develop a Vision-Language Transformer (VLT) framework for enhanced referring segmentation.
• To address the limitations of fixed queries in transformers for dynamic language understanding.
• To improve the holistic understanding of vision-language features for accurate object segmentation.

Main Methods:

• Proposed a Query Generation Module to dynamically produce input-specific queries for diverse language comprehensions.
• Introduced a Query Balance Module to selectively fuse responses from generated queries for optimal mask generation.
• Implemented inter-sample learning with masked contrastive learning to improve handling of varied language expressions for the same object.

Main Results:

• The proposed VLT framework achieves state-of-the-art results in referring segmentation.
• The approach demonstrates consistent performance improvements across five benchmark datasets.
• The framework is lightweight, offering an efficient solution for complex vision-language tasks.

Conclusions:

• The novel VLT framework effectively addresses the challenge of dynamic language understanding in referring segmentation.
• Dynamic query generation and fusion mechanisms significantly enhance segmentation accuracy.
• The proposed methods provide a robust and efficient solution for state-of-the-art referring segmentation.