HAFormer: Unleashing the Power of Hierarchy-Aware Features for Lightweight Semantic Segmentation

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Convolutional Neural Networks (CNNs) and Transformers excel in semantic segmentation.
• Integrating CNNs and Transformers captures local and global context.
• Resource constraints limit the application of complex models.

Purpose of the Study:

• Introduce HAFormer, a lightweight semantic segmentation model.
• Address computational resource limitations in deep learning models.
• Enhance semantic segmentation accuracy with efficient architectures.

Main Methods:

• Developed HAFormer, integrating CNNs' hierarchical feature extraction with Transformers' global dependency modeling.
• Designed Hierarchy-Aware Pixel-Excitation (HAPE) for multi-scale local feature extraction.
• Implemented an Efficient Transformer (ET) module to reduce quadratic complexity.
• Utilized a correlation-weighted Fusion (cwF) module for merging diverse feature representations.

Main Results:

• Achieved 74.2% mIoU on Cityscapes and 71.1% mIoU on CamVid test datasets.
• Demonstrated high performance with minimal computational overhead and compact model size.
• Reached frame rates of 105FPS (Cityscapes) and 118FPS (CamVid) on a single 2080Ti GPU.
• Source code is publicly available.

Conclusions:

• HAFormer offers a compelling solution for lightweight semantic segmentation.
• The model balances high accuracy with computational efficiency.
• HAFormer is suitable for applications with limited computational resources.