One-Stage Visual Relationship Referring With Transformers and Adaptive Message Passing.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Visual relationship referring (VRR) involves localizing subject and object entities within an image based on a given relationship query.
• Existing VRR methods are categorized as one-stage (direct localization, lower accuracy) or multi-stage (indirect, proposal-based, better accuracy).

Purpose of the Study:

• To propose a novel one-stage visual relationship referring approach (VRR-TAMP) that enhances prediction accuracy.
• To formulate VRR as an end-to-end bounding box regression problem.

Main Methods:

• VRR-TAMP integrates Transformers and an adaptive message passing mechanism for improved visual relationship referring.
• It encodes image and relationship queries into embeddings, processes them via a cross-modal Transformer encoder, and uses an entity-specific information distiller (SR-GMP) for adaptive distillation.

Main Results:

• The proposed VRR-TAMP approach significantly outperforms existing one-stage VRR methods.
• It achieves competitive results compared to state-of-the-art multi-stage VRR methods on VRD and Visual Genome datasets.

Conclusions:

• The novel one-stage VRR-TAMP method offers a more accurate and efficient solution for visual relationship referring.
• Integrating Transformers with adaptive message passing is effective for improving entity localization in complex visual relationships.