Collect-Then-Select: Multispectral Detection Transformer With Collaborative Aggregation and Progressive Refinement

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Object detection commonly uses Convolutional Neural Network (CNN) or Detection Transformer (DETR) architectures.
• Current Red-Green-Blue (RGB)-Infrared (IR) object detection methods often use a
• Collect-while-Select
• (CWS) strategy, which can lead to interference from irrelevant data.

Purpose of the Study:

• To develop an improved object detection framework for RGB-IR data.
• To address limitations in current methods regarding multimodal feature refinement and query selection.

Main Methods:

• Proposed the Customized Multispectral Detection Transformer (CS-MDETR) with a
• Collect-then-Select
• (CTS) strategy.
• Introduced Dynamic Differential Compensation (DDC) and Spatial-Channel Consistency Constraint (S3C) modules for feature compensation and consistency.
• Implemented Salience-CIoU-aware Query Selection (SCQS) and Geometric Relation Encoding (GRE) for enhanced query selection and prediction consistency.

Main Results:

• CS-MDETR with the CTS strategy effectively integrates cross-modal information.
• The model demonstrates superior performance in refining multimodal features compared to existing methods.
• Achieved state-of-the-art results on six challenging benchmarks.

Conclusions:

• The proposed CS-MDETR and CTS strategy significantly enhance RGB-IR object detection.
• The novel modules and strategies effectively mitigate interference and improve detection accuracy.
• CS-MDETR represents a significant advancement over current CNN-based and DETR-based RGB-IR detectors.