OLIVE: Object Level In-Context Visual Embeddings.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Generalist vision-language models (VLMs) show strong multimodal reasoning but lack fine-grained object understanding and grounding.
• Current VLMs align text and image tokens at a patch level, leading to inefficient embedding alignment and inclusion of background noise.
• Existing models struggle with generalization to new visual concepts and require extensive fine-tuning for domain-specific tasks.

Purpose of the Study:

• To develop a novel method for controllable object-level reasoning in vision-language models.
• To enhance the fine-grained understanding and grounding capabilities of VLMs.
• To improve the efficiency and adaptability of VLMs for domain-specific applications.

Main Methods:

• Prompting large language models (LLMs) with in-context visual object vectors.
• Eliminating the need to fuse extensive image patch features for faster training.
• Implementing region-level retrieval using object representations for rapid adaptation.

Main Results:

• Achieved competitive performance in referring object classification and captioning.
• Demonstrated zero-shot generalization capabilities to unseen visual concepts.
• Showcased robustness in visually challenging contexts without additional training.

Conclusions:

• The proposed method enables controllable object-level reasoning by leveraging visual object vectors.
• This approach significantly improves training efficiency and adaptability for VLMs.
• The method offers enhanced generalization and robustness, addressing key limitations of current VLM architectures.