Cross-modality interaction for few-shot multispectral object detection with semantic knowledge.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Multispectral object detection (MOD) enhances object detection (OD) by integrating thermal and visible imagery, improving robustness in varied lighting.
• Current MOD techniques require extensive annotated datasets, limiting their practical application.
• Few-shot learning principles offer a potential solution to reduce data dependency.

Purpose of the Study:

• To introduce and address the novel task of few-shot multispectral object detection (FSMOD).
• To develop a method capable of performing MOD with minimal annotated data per category.
• To enhance the efficiency and applicability of multispectral object detection systems.

Main Methods:

• A cross-modality interaction (CMI) module was designed to fuse information from visible and thermal modalities using attention mechanisms during feature extraction.
• Modality-specific backbone features with enhanced discrimination were extracted, guided by the CMI module.
• A semantic prototype metric (SPM) loss was developed, incorporating word embeddings to stabilize category representation in low-data scenarios.

Main Results:

• The proposed FSMOD method demonstrated state-of-the-art performance on a customized FSMOD dataset.
• The CMI module effectively improved the discrimination of modality-specific features.
• The SPM loss enhanced the model's ability to generalize from limited visual data by leveraging semantic knowledge.

Conclusions:

• The developed FSMOD approach significantly reduces the annotation burden for multispectral object detection.
• The integration of cross-modality interaction and semantic information is effective for few-shot learning in MOD.
• This work paves the way for more data-efficient and robust object detection systems in complex environments.