CMFX: Cross-modal fusion network for RGB-X crowd counting

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Existing crowd counting methods often combine RGB images with complementary X-modalities for improved accuracy.
• A key challenge is developing models adaptable to diverse sensors due to inter-modal feature variations.

Purpose of the Study:

• To propose CMFX, a unified fusion framework for robust RGB-X crowd counting.
• To address the limitations of sensor-specific models in multimodal crowd counting.

Main Methods:

• CMFX incorporates a Fast Feature Aggregation Module (FFAM) using lightweight mixed attention for low-level feature fusion.
• A Cross-Modal Feature Interaction Module (CFIM) rectifies and fuses high-level features by exploring correlations.
• A Cross-Modal Feature Decoding Module (CFDM) utilizes graph convolution for refining cross-modal features.

Main Results:

• CMFX was evaluated on RGBT-CC, DroneRGBT, and ShanghaiTechRGBD datasets, unifying depth and thermal modalities with RGB.
• The framework demonstrated excellent performance in both RGB-depth and RGB-thermal crowd counting scenarios.
• The proposed modules effectively enhance multimodal feature representation and interaction.

Conclusions:

• CMFX provides a unified and adaptable solution for RGB-X crowd counting.
• The framework overcomes challenges related to sensor differences and achieves state-of-the-art results.
• This work paves the way for more generalized multimodal crowd analysis systems.