Related Experiment Videos
M3R++: Masked Token Mixup and Dual Cross-Modal Masking for Zero-Shot Learning
IEEE Transactions on Pattern Analysis and Machine Intelligence
|August 14, 2026
Summary
This study introduces M3R++, a novel framework for zero-shot learning (ZSL) that improves recognition of unseen classes by enhancing visual-semantic connections. M3R++ reduces bias towards seen classes, leading to more accurate object recognition in ZSL tasks.
Area of Science:
- Computer Science
- Artificial Intelligence
- Machine Learning
Background:
- Zero-shot learning (ZSL) aims to classify objects from classes not encountered during training.
- Existing ZSL methods struggle with misclassifying unseen classes due to weak visual-semantic correspondences, leading to bias towards seen classes.
Purpose of the Study:
- To propose M3R++, a novel framework designed to enhance zero-shot learning by establishing robust connections between semantic attributes and visual features.
- To reduce the bias towards seen classes in ZSL by improving the discrimination between seen and unseen class samples.
Main Methods:
- The M3R++ framework incorporates four key components: Random Token Mixup (RTM), Unseen Class Detection (UCD), Dual Cross-modality Masked auto-encoders (DCM), and Diverse Prompt Learning (DPL).
- RTM generates synthetic unseen class samples by mixing visual tokens and semantic attributes.
- UCD uses entropy values to differentiate seen and unseen classes, DCM employs cross-modal masking for robust feature relationships, and DPL uses divergence loss for diverse visual prompts.
Main Results:
- Extensive experiments were conducted on four challenging datasets to evaluate the M3R++ model's performance.
- The proposed M3R++ model demonstrated significant effectiveness and superiority over existing methods in zero-shot learning tasks.
- The framework successfully built stronger connections between semantic attributes and visual clues, effectively reducing the bias towards seen classes.
Conclusions:
- The M3R++ framework offers a significant advancement in zero-shot learning by effectively addressing the challenge of misclassification and bias towards seen classes.
- The combination of masked token mixup and dual cross-modal masking provides a robust approach for transferring knowledge from seen to unseen classes.
- M3R++ shows great potential for improving the accuracy and reliability of object recognition in scenarios with limited or no prior exposure to specific classes.
