Multimodal Image Representation Learning With Limited Visual-Tactile Data.

Area of Science:

• Computer Vision
• Machine Learning
• Robotics

Background:

• Multimodal visual-tactile image representation learning (VTL) excels with large datasets but struggles with limited data.
• Existing methods often fail to focus on discriminative features, leading to performance degradation in low-data scenarios.

Purpose of the Study:

• Introduce visual-tactile image representation learning with limited data (VTL-L) for practical applications.
• Propose a novel network (MOA-Net) to address limited data and modality discrepancy challenges in VTL-L.

Main Methods:

• Multi-order feature enhancement (MFE) module to aggregate topological information and obtain discriminative features.
• Alignment-free visual-tactile fusion (AVTF) module for cross-modality fusion without alignment, mitigating modality discrepancy.
• Dual counterfactual intervention (DCI) loss to optimize fused feature and probability distributions.

Main Results:

• MOA-Net demonstrates superior performance across various tasks and datasets under limited-data conditions.
• The MFE module effectively reduces attention noise and enhances feature representation.
• The AVTF module successfully fuses features across modalities without alignment.

Conclusions:

• The proposed MOA-Net and VTL-L task offer a promising direction for robust multimodal learning with limited data.
• The method effectively handles modality discrepancy and enhances feature discriminability in low-resource settings.