Prophet: Prompting Large Language Models With Complementary Answer Heuristics for Knowledge-Based Visual Question...

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Knowledge-based visual question answering (VQA) traditionally relies on explicit knowledge bases, often leading to performance limitations due to irrelevant information.
• Recent approaches utilize large language models (LLMs) as implicit knowledge engines, but their full potential is untapped due to insufficient visual context in prompts.

Purpose of the Study:

• To introduce Prophet, a novel framework designed to optimize LLM prompting for knowledge-based VQA by incorporating answer heuristics.
• To enhance the ability of LLMs to leverage visual information and external knowledge for more accurate VQA.

Main Methods:

• Train a base VQA model without external knowledge.
• Extract complementary answer heuristics (candidates and answer-aware examples) from the trained VQA model.
• Encode these heuristics into a formatted prompt to guide a large language model (LLM) in answering VQA questions.

Main Results:

• Prophet, when integrated with GPT-3, significantly surpasses existing state-of-the-art methods on four challenging knowledge-based VQA datasets.
• The framework demonstrates generality, supporting various VQA models and LLMs, including commercial and open-source options.

Conclusions:

• Prophet offers a flexible and effective approach to improve knowledge-based VQA by enhancing LLM's understanding through heuristic-guided prompting.
• The Prophet++ extension shows potential for further advancements by integrating with large multimodal models.