Segment as Points for Efficient and Effective Online Multi-Object Tracking and Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current multi-object tracking and segmentation (MOTS) methods struggle with ambiguous instance embeddings due to large receptive fields in convolutional neural networks.
• This ambiguity leads to difficulties in accurately associating objects across frames, particularly in crowded scenes.

Purpose of the Study:

• To develop a novel method for learning robust instance embeddings for MOTS by leveraging point cloud representations.
• To improve the accuracy and efficiency of multi-object tracking and segmentation, addressing limitations of existing tracking-by-detection approaches.

Main Methods:

• Proposed a method that converts image representations into un-ordered 2D point clouds to exploit the non-overlapping nature of instance segments.
• Enriched point-wise features by formulating multiple data modalities into point-wise representations.
• Learned instance embeddings using foreground point clouds, environment point clouds, and the smallest circumscribed bounding box.
• Modified the SpatialEmbedding method for instance segmentation and developed the PointTrackV2 framework.

Main Results:

• PointTrackV2 achieved state-of-the-art performance, outperforming existing methods by significant margins (e.g., 4.8% higher sMOTSA for pedestrians).
• The framework operates at near real-time speeds (20 FPS on a single 2080Ti), demonstrating practical utility.
• Extensive evaluations on three datasets confirmed the method's effectiveness and efficiency.

Conclusions:

• The proposed point cloud-based approach effectively resolves ambiguities in instance embeddings, leading to superior MOTS performance.
• PointTrackV2 offers a highly effective and efficient solution for multi-object tracking and segmentation.
• Introduced the APOLLO MOTS dataset to address the lack of crowded car scenes in existing benchmarks, facilitating further research.