Object-Centric Representation Learning for Video Scene Understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Depth-aware Video Panoptic Segmentation (DVPS) is complex, requiring pixel-wise semantic class, 3D depth, and object tracking across frames.
• Current methods often treat these as independent tasks, limiting the exploitation of inter-task relationships and requiring extensive parameter tuning.

Purpose of the Study:

• To introduce Slot-IVPS, an object-centric approach for unified object representations in DVPS.
• To enable simultaneous capture of semantic and depth information for all video objects, including background and foreground.

Main Methods:

• Developed Integrated Panoptic Slots (IPS) for unified semantic and depth representation.
• Proposed an integrated feature generator/enhancer for depth-aware features.
• Introduced the Integrated Video Panoptic Retriever (IVPR) for retrieving and encoding spatial-temporal coherent object features into IPS.

Main Results:

• Achieved state-of-the-art performance on both Depth-aware Video Panoptic Segmentation and Video Panoptic Segmentation tasks.
• Demonstrated the effectiveness of unified object-centric representations for DVPS.
• IPS representations were successfully decoded into depth maps, classifications, masks, and object instance IDs.

Conclusions:

• Slot-IVPS offers a novel and effective solution for DVPS by leveraging unified object-centric representations.
• The proposed IPS representation and IVPR facilitate integrated semantic and depth information processing.
• The method shows significant advancements in video understanding tasks requiring both semantic and geometric scene interpretation.