Full-spectrum prompt tuning with sparse MoE for open-set recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Open-set recognition (OSR) advances often focus on high-level visual features from vision-language models (VLMs).
• Low-level visual details from shallow image encoder layers are underutilized in current OSR methods.
• Integrating low-level features into high-level textual prompts presents representation challenges.

Purpose of the Study:

• To propose a novel method, Full-Spectrum Prompt Tuning with Sparse Mixture-of-Experts (FSMoE), for enhancing open-set recognition.
• To leverage full-spectrum visual features across VLM image encoder layers for improved textual prompt generation.
• To address the challenge of integrating low-level visual details into prompts for better unknown class identification.

Main Methods:

• FSMoE utilizes full-spectrum visual features from VLMs to enhance textual prompts.
• Two groups of textual tokens (high-level and low-level) interact with corresponding visual features.
• A sparse Mixture-of-Experts mechanism adaptively selects and weights low-level visual features.
• Routing consistency contrastive loss enforces intra-class consistency among experts.

Main Results:

• The proposed FSMoE method effectively enhances textual prompts using both high-level and low-level visual features.
• The sparse Mixture-of-Experts mechanism successfully mitigates redundancy in low-level visual details.
• Experimental results validate the effectiveness of FSMoE in open-set recognition tasks.

Conclusions:

• FSMoE offers a comprehensive approach to open-set recognition by integrating full-spectrum visual information into textual prompts.
• The method overcomes the limitations of solely relying on high-level features and addresses feature representation disparities.
• FSMoE demonstrates significant potential for advancing the field of open-set recognition using vision-language models.