Negative prompt-guided optimization: Enhancing soft prompt generalization in vision-language models

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Prompt learning is key for adapting vision and language models to new tasks.
• Existing methods overfit training data, hurting performance on unseen classes.
• Failure to align negative prompts contributes to this overfitting problem.

Purpose of the Study:

• To introduce Negative Prompt-Guided Optimization (NPGO) to mitigate overfitting in prompt learning.
• To enhance the generalization capabilities of vision and language models on unseen classes.
• To improve representation learning and inference stability.

Main Methods:

• Utilized adversarial training with prompts containing negative text.
• Introduced a negative adversarial loss to encourage uniform probability distribution between positive and negative prompts.
• Empirically analyzed existing methods' failure to align negative prompts.

Main Results:

• NPGO significantly alleviates misalignment issues present in current methods.
• Demonstrated remarkable improvements in generalization performance for unseen classes.
• Achieved superior representation learning and inference stability across 11 diverse datasets.

Conclusions:

• NPGO offers a robust solution to overfitting in prompt learning.
• The method enhances model adaptability and performance on novel data.
• Negative prompt alignment is crucial for effective generalization in vision and language models.