Benchmarking Semantic Segmentation Models via Appearance and Geometry Attribute Editing

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Semantic segmentation is crucial for applications like autonomous driving and medical imaging.
• Testing segmentation models in diverse, complex scenarios is essential for practical deployment.
• Existing evaluation methods often focus narrowly on global transformations, neglecting object-level variations.

Purpose of the Study:

• To develop an automatic data generation pipeline (Gen4Seg) for stress-testing semantic segmentation models.
• To investigate the impact of both appearance and geometric attribute variations on model performance.
• To create new benchmarks (Pascal-EA, COCO-EA) for evaluating model robustness.

Main Methods:

• Utilizing diffusion models to edit visual attributes of real images while preserving structural information.
• Generating challenging samples with controlled variations in object color, material, size, position, and image-level factors (weather, style).
• Reusing existing segmentation labels for generated images to reduce data construction costs.

Main Results:

• Advanced open-vocabulary models showed no superior robustness against geometric variations compared to closed-set methods.
• Traditional data augmentation techniques (CutOut, CutMix) demonstrated limited effectiveness in improving robustness against appearance variations.
• The Gen4Seg pipeline, when used for data augmentation, enhanced both in-distribution and out-of-distribution performance of segmentation models.

Conclusions:

• Generative models show significant potential for automated analysis and stress-testing of semantic segmentation models.
• The findings highlight limitations in current advanced models and augmentation techniques regarding robustness.
• The developed pipeline and benchmarks can aid researchers in creating more reliable and robust semantic segmentation solutions.