Open-set Anomaly Segmentation in Complex Scenarios | JoVE Visualize

Area of Science:

Computer Vision
Machine Learning
Autonomous Systems

Background:

Accurate segmentation of out-of-distribution (OoD) objects, or anomalies, is critical for safety-critical applications like autonomous driving.
Existing benchmarks for anomalous segmentation primarily evaluate performance under ideal weather conditions, failing to address real-world challenges such as low illumination, fog, and heavy rain.
This oversight leads to untrustworthy evaluations and highlights safety risks in open-set environments.

Purpose of the Study:

To introduce ComSAmy, a novel benchmark for complex scenario anomaly segmentation, designed to evaluate model performance under diverse adverse weather and driving conditions.
To assess the limitations of current state-of-the-art anomalous segmentation models in realistic, challenging open-world scenarios.
To propose a new method, Energy-Entropy Learning (EEL) and a diffusion-based data synthesizer, to enhance anomaly segmentation robustness.

Main Methods:

Development of the ComSAmy benchmark, featuring a wide range of adverse weather, dynamic environments, and anomaly types.
Evaluation of existing anomalous segmentation models on the ComSAmy benchmark to identify performance gaps.
Proposal of a novel Energy-Entropy Learning (EEL) strategy to leverage complementary energy and entropy information.
Introduction of a diffusion-based synthesizer for generating diverse and high-quality anomalous training data.

Main Results:

State-of-the-art anomalous segmentation models exhibit significant deficiencies in complex scenarios, posing safety risks.
The proposed diffusion-based synthesizer effectively generates diverse and high-quality anomalous images.
The integrated DiffEEL framework significantly enhances existing models, demonstrating effectiveness and generalizability.
Average improvements of 4.96% in AUPRC and 9.87% in FPR95 were observed on public and ComSAmy benchmarks.

Conclusions:

Current anomalous segmentation methods are insufficient for safe deployment in real-world, adverse conditions.
The ComSAmy benchmark provides a more realistic evaluation of model performance.
The proposed DiffEEL framework offers a robust and generalizable solution for improving anomaly segmentation in complex environments.
The developed methods contribute to safer and more reliable autonomous driving systems.