M3R++: Masked Token Mixup and Dual Cross-Modal Masking for Zero-Shot Learning

Zero-shot learning (ZSL) aims to recognize novel class objects by transferring semantic knowledge from seen to unseen classes. However, existing methods often misclassify unseen class images as seen classes due to a lack of robust correspondences between visual features and semantic attributes. To address these issues, we propose M3R++, a novel framework that utilizes masked token mixup and dual cross-modal masking to enhance zero-shot learning by building strong connections between semantic attributes and visual clues, thereby reducing the bias towards seen classes. The M3R++ framework consists of four key components: Random Token Mixup (RTM), Unseen Class Detection (UCD), Dual Cross-modality Masked auto-encoders (DCM), and Diverse Prompt Learning (DPL). First, RTM can broaden the representation space of unseen classes by mixing visual tokens from different class samples and combining their semantic attributes to generate synthetic unseen class samples. Second, UCD adaptively assigns different entropy values to distinguish between seen and unseen class samples, reducing classification bias. Third, DCM can establish robust relationships between semantic attributes and visual knowledge through vision guided and attribute-guided cross-modal masking. Fourth, DPL introduces a divergence loss to encourage diversity among a set of visual prompts, thereby providing rich visual clues for accurately mining correspondences between semantic attributes and visual knowledge. Extensive experiments on four challenging datasets demonstrate the effectiveness and superiority of the proposed M3R++ model.