Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Cross-modality recognition bridges data from different sources (e.g., images, spectra) for tasks in science and security.
• Existing methods struggle with information loss or reliance on difficult explicit modality transfer.
• A robust approach is needed to handle modality gaps effectively.

Purpose of the Study:

• To develop a novel neural network module (cmUNet) for learning modality-agnostic representations.
• To propose MarrNet, a system integrating cmUNet for superior cross-modality matching.
• To enhance robustness against occlusions and disguises in cross-modality recognition.

Main Methods:

• Proposed a compact encoder-decoder neural module (cmUNet) for modality-agnostic representation learning.
• Employed cross-modality transformation and in-modality reconstruction with adversarial/perceptual loss.
• Integrated cmUNet into MarrNet for cross-modality matching tasks, outputting similarity scores.

Main Results:

• MarrNet demonstrated superior performance on Raman-infrared spectrum matching, person re-identification, and heterogeneous face recognition.
• The method achieved over 10% improvement compared to state-of-the-art techniques.
• MarrNet exhibited excellent robustness against occlusions and disguises, indicating effective modality gap bridging.

Conclusions:

• The proposed cmUNet is a versatile building block for various cross-modality applications.
• MarrNet offers a robust and high-performing solution for cross-modality recognition challenges.
• Robustness to occlusions is a key indicator of successful modality gap bridging.