Prompt-Based Modality Alignment for Effective Multi-Modal Object Re-Identification.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Multi-modal Object Re-Identification (ReID) faces challenges in aggregating complementary information, especially under varying illumination.
• Current methods often use complex, coupled architectures leading to high computational costs and struggle with distribution gaps across different spectra.
• Effective joint representation of multi-modal features remains a significant hurdle in ReID research.

Purpose of the Study:

• To propose PromptMA, a novel framework for effective communication and aggregation of complementary information across different modalities in ReID.
• To bridge the distribution gap between different image spectra for improved multi-modal feature representation.
• To develop a flexible ReID method capable of handling missing modalities.

Main Methods:

• Injecting learnable multi-modal prompts into the Image Encoder to facilitate inter-modal communication.
• Implementing a prompt exchange mechanism for alternating interaction between prompts and modal token embeddings.
• Utilizing Prompt-based Token Selection (PBTS) and Prompt-based Modality Fusion (PBMF) modules for efficient feature fusion and background interference reduction.

Main Results:

• PromptMA achieves state-of-the-art performance on four benchmark datasets.
• Demonstrated significant improvements, surpassing current benchmarks by over 15% on MSVR310 and 6% on RGBNT201.
• The framework effectively mitigates illumination issues and bridges spectral distribution gaps.

Conclusions:

• PromptMA offers an effective and computationally efficient solution for multi-modal Object Re-Identification.
• The prompt-based approach enhances feature aggregation and fusion, leading to superior ReID performance.
• The method's flexibility makes it suitable for real-world scenarios, including those with missing modalities.