Monocular Multi-Object 3D Visual Language Tracking

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Existing Visual Language Tracking (VLT) methods are limited to 2D or single-object 3D tracking.
• Current 3D multi-object tracking relies on sensor data lacking language descriptions.
• Redundant language descriptions hinder precise multi-object localization in VLT.

Purpose of the Study:

• To extend Visual Language Tracking (VLT) to multi-object 3D tracking using monocular video.
• To introduce a novel framework, dataset, and neural model for this task.
• To enable robust 3D tracking of multiple objects from monocular video using language guidance.

Main Methods:

• Introduced the Monocular Multi-object 3D Visual Language Tracking (MoMo-3DVLT) task and dataset (MoMo-3DRoVLT).
• Developed MoMo-3DVLTracker, a neural model with multimodal feature extraction and a language encoder-decoder.
• Integrated a differentiable linked-memory mechanism with depth-guided, language-conditioned reasoning.

Main Results:

• The proposed MoMo-3DVLTracker achieves state-of-the-art performance on the MoMo-3DRoVLT dataset.
• Demonstrated superior multi-object 3D tracking accuracy using monocular video compared to existing methods.
• The custom dataset provides extensive 3D annotations and natural language descriptions for VLT research.

Conclusions:

• This work presents the first successful extension of VLT to multi-object 3D tracking with monocular video.
• The developed framework, dataset, and model establish a new baseline for this challenging task.
• The approach offers a promising direction for real-world applications requiring language-guided 3D object tracking.