Related Experiment Video
Updated: Sep 8, 2025
04:48
Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique
Published on: July 5, 2024
491
Deformation-Resilient Multigranularity Learning for Unaligned RGB-T Semantic Segmentation
Summary
This study introduces a new benchmark and method for unaligned RGB-Thermal semantic segmentation, improving object mask accuracy by aligning features across modalities. The deformation-resilient multigranularity learning (DML) method effectively handles misaligned image pairs.
Area of Science:
- Computer Vision
- Machine Learning
- Image Processing
Background:
- Semantic segmentation (SS) typically requires aligned RGB-Thermal (RGB-T) image pairs.
- Real-world RGB-T data is often unaligned, posing significant challenges for existing SS methods.
- Pixel-level alignment of unaligned RGB-T images is computationally intensive and difficult.
Purpose of the Study:
- To address the challenge of unaligned RGB-T image pairs in semantic segmentation.
- To introduce a novel benchmark dataset for unaligned RGB-T SS.
- To propose a deformation-resilient multigranularity learning (DML) method for robust RGB-T SS.
Main Methods:
- Developed a new unaligned RGB-T SS benchmark dataset.
- Proposed the deformation-resilient multigranularity learning (DML) method.
- Introduced a deformation-aware complementary feature enhancer (DCFE) with deformation-aware feature alignment (DFA) and complementary feature aggregation (CFA) modules.
- Designed a multigranularity mask refinement engine (MMFE) incorporating class-agnostic saliency prediction (CSP) and class-aware edge generation (CEG).
Main Results:
- The DML method effectively aligns multimodal features in a coarse-to-fine strategy, mitigating interference from warped modalities.
- DFA estimates deformation fields for improved spatial alignment, while CFA aggregates complementary contextual information across scales.
- MMFE enhances semantic alignment and inter-class separability, producing object masks with sharp boundaries.
- Experiments demonstrated superior performance of DML on both aligned and unaligned datasets compared to existing methods.
Conclusions:
- The proposed DML method offers a robust solution for semantic segmentation using unaligned RGB-T image pairs.
- The new benchmark facilitates research on handling real-world, misaligned multimodal imaging data.
- DML achieves state-of-the-art results, highlighting the importance of addressing modality alignment in RGB-T SS.
