Scale-Invariant Feature Matching Network for V-D-T Few-Shot Semantic Segmentation

Area of Science:

• Computer Vision
• Machine Learning
• Image Processing

Background:

• Few-shot semantic segmentation (FSS) requires dense prediction from limited annotated samples across multiple image modalities.
• Existing methods often treat diverse modalities (visible, depth, thermal) equally, neglecting their inherent differences.
• Challenges include significant object size variations and ineffective support-query connections in current matching paradigms.

Purpose of the Study:

• To propose a novel scale-invariant feature matching network (SFM-Net) for visible-depth-thermal (V-D-T) few-shot semantic segmentation.
• To address limitations in handling multi-modal differences and scale variations in FSS.
• To improve the accuracy and robustness of semantic segmentation with few annotated samples.

Main Methods:

• SFM-Net integrates an encoder, feature matching block, feature elevation block, and decoder.
• A pixel-to-patch cross-attention (PTPCA) module with pixel-to-patch pooling (PTP-pool) establishes scale-invariant relationships.
• A prior-related fusion (PF) module and reverse attention (RA) unit enhance feature integration and final segmentation.

Main Results:

• The model effectively fuses features from visible, depth, and thermal images, accounting for modality differences.
• Scale-invariant relationships are established, improving segmentation for objects of varying sizes.
• Experiments on the VDT-2048-5i dataset demonstrate superior performance over state-of-the-art methods.

Conclusions:

• The proposed SFM-Net significantly advances V-D-T few-shot semantic segmentation.
• The novel modules effectively handle multi-modal fusion and scale variations.
• The approach offers a robust solution for dense prediction tasks with limited data.