MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Existing semantic segmentation datasets suffer from inconsistent taxonomies and annotation practices, hindering cross-domain model performance.
• A naive merge of diverse datasets leads to suboptimal results due to these inconsistencies.

Purpose of the Study:

• To develop a unified composite dataset (MSeg) for training robust semantic segmentation models.
• To enable models to generalize effectively across different domains and unseen datasets.

Main Methods:

• Reconciled taxonomies and aligned pixel-level annotations by relabeling over 220,000 object masks across 80,000 images.
• Utilized zero-shot cross-dataset transfer to benchmark model robustness.
• Trained and evaluated semantic, instance, and panoptic segmentation models.

Main Results:

• MSeg training yielded substantially more robust models compared to training on individual or naively merged datasets.
• A model trained on MSeg achieved first place on the WildDash-v1 leaderboard for robust semantic segmentation without prior exposure.
• The MSeg-trained model secured second place in the 2020 Robust Vision Challenge (RVC) despite differing evaluation taxonomies.

Conclusions:

• The MSeg dataset enables the training of single semantic segmentation models that perform effectively across domains and generalize to unseen data.
• MSeg significantly improves model robustness and cross-dataset transfer capabilities.
• The comprehensive evaluation using MSeg provides crucial insights for advancing robust, efficient, and complete scene understanding.