ETFT: Equiangular Tight Frame Transformer for Imbalanced Semantic Segmentation.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Semantic segmentation faces challenges with class imbalance, where label distribution is uneven.
• Existing methods use neural collapse and Equiangular Tight Frames (ETF) to improve minor class discrimination, but struggle with class correlation.
• Current approaches result in fixed classifiers, limiting adaptability to diverse input images.

Purpose of the Study:

• To propose a novel transformer-based model, the Equiangular Tight Frame Transformer (ETFT), for imbalanced semantic segmentation.
• To dynamically generate classifiers as a function of input, balancing class discrimination and correlation.
• To enhance the adaptability and performance of semantic segmentation models on datasets with uneven class distributions.

Main Methods:

• The ETFT model jointly processes features and classifiers using an ETF structure within a transformer architecture.
• Input patch tokens and the ETF-initialized classifier are processed together during the attention mechanism.
• The classifier is dynamically adjusted based on input correlations and combined with a fixed ETF classifier.

Main Results:

• The ETFT model achieves improved discriminability between classes while preserving contextual correlation.
• The dynamically generated classifier adapts to input, enhancing performance.
• Experiments show the proposed method outperforms state-of-the-art approaches on ADE20K and Cityscapes datasets for imbalanced semantic segmentation.

Conclusions:

• The ETFT model effectively addresses the challenge of class imbalance in semantic segmentation.
• Dynamic classifier generation offers a significant advantage over fixed classifiers for varying input data.
• The proposed approach demonstrates superior performance and robustness in imbalanced semantic segmentation tasks.