Task-to-Instance Prompt Learning for Vision-Language Models at Test Time.

Area of Science:

• Computer Vision
• Natural Language Processing
• Machine Learning

Background:

• Prompt learning adapts pre-trained vision-language models (VLMs) by tuning trainable tokens.
• Existing methods often require extra annotated data for training.
• Test-time prompt learning methods typically learn prompts per sample, neglecting task-level knowledge.

Purpose of the Study:

• To develop a novel test-time prompt learning method for VLMs using only unlabeled data.
• To leverage both task-level and instance-level knowledge for improved VLM adaptation.
• To address the limitations of existing methods that rely on annotated data or single-sample prompt learning.

Main Methods:

• Propose Task-to-Instance Prompt Learning (TIPPLE), a two-stage training strategy.
• Utilize an online pseudo-labeling paradigm with an auxiliary text classification task.
• Incorporate a diversity regularization term for task-oriented prompt learning.
• Combine a learned task-level prompt with a tunable residual for instance-level adaptation.

Main Results:

• TIPPLE demonstrates superior performance on 15 downstream datasets.
• Achieved an average improvement of 1.87% over state-of-the-art methods.
• Validated using the ViT-B/16 visual backbone.

Conclusions:

• TIPPLE effectively adapts VLMs using unlabeled test data by integrating task- and instance-level knowledge.
• The proposed method outperforms existing test-time prompt learning approaches.
• The open-sourced code facilitates further research and application.