VLExpan: A visual-enhanced LLM framework with inductive and deductive policies for entity set expansion

Area of Science:

• Natural Language Processing
• Computer Vision
• Knowledge Acquisition

Background:

• Existing Entity Set Expansion (ESE) methods primarily use textual information, limiting fine-grained expansion and recall of less common entities.
• Traditional bootstrap frameworks for ESE are prone to error propagation, impacting overall performance.
• There is a need for ESE methods that leverage multimodal information for improved accuracy and robustness.

Purpose of the Study:

• To propose a Visual-enhanced LLM framework with inductive and deductive policies (VLExpan) for improved Entity Set Expansion.
• To address the limitations of text-only ESE methods, including fine-grained expansion and error propagation.
• To enhance the recall of long-tail entities through the integration of visual data.

Main Methods:

• VLExpan integrates visual information using a vision-language model for iterative seed entity expansion.
• A Large Language Model (LLM) is employed to induce class names from seed entities.
• A deductive policy refines the expansion process using the induced class name and LLM capabilities.

Main Results:

• VLExpan demonstrated significant improvements in Entity Set Expansion performance.
• The framework achieved average score improvements of 3.36% (MAP@10) and 4.51% (MAP@20/MAP@50) on benchmark datasets.
• Experimental results validate the effectiveness of incorporating visual information and LLMs in ESE.

Conclusions:

• VLExpan offers a superior approach to Entity Set Expansion by effectively combining visual and textual data with LLMs.
• The proposed framework mitigates error propagation inherent in traditional methods.
• VLExpan advances the field of knowledge acquisition by enabling more accurate and comprehensive entity set discovery.