Dense Pixel-Level Interpretation of Dynamic Scenes With Video Panoptic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Understanding dynamic scenes is crucial for real-world applications like autonomous driving and augmented reality.
• Existing methods lack comprehensive analysis of spatio-temporal scene dynamics.
• A unified benchmark and evaluation metric are needed for video panoptic segmentation.

Purpose of the Study:

• Introduce a new benchmark, Video Panoptic Segmentation (VPS), for dynamic scene understanding.
• Present novel datasets (Cityscapes-VPS, VIPER) and an evaluation metric (video panoptic quality - VPQ).
• Develop an advanced network (VPSNet++) for simultaneous classification, detection, segmentation, and tracking in videos.

Main Methods:

• Propose VPSNet++, an enhanced top-down panoptic segmentation network with pixel-level feature fusion and object-level association.
• Incorporate auxiliary tasks: panoptic boundary learning and instance discrimination learning.
• Utilize spatio-temporally clustered pixel embeddings for improved segmentation and tracking.

Main Results:

• VPSNet++ significantly outperforms the baseline FuseTrack (default VPSNet).
• Achieved state-of-the-art performance on both Cityscapes-VPS and VIPER datasets.
• Demonstrated effective simultaneous tracking and segmentation of all video identities.

Conclusions:

• The proposed VPS benchmark, datasets, and metric facilitate advancements in dynamic scene understanding.
• VPSNet++ represents a significant step forward in video panoptic segmentation.
• Publicly available resources will foster further research and development in the field.