Optimizing document retrieval using massive text embeddings and LLM prompt engineering

Area of Science:

• Information Science
• Artificial Intelligence
• Computational Linguistics

Background:

• The exponential growth of digital data, particularly scientific publications, complicates efficient information retrieval.
• Manual literature reviews are increasingly time-consuming due to the sheer volume of published research.
• Large language models (LLMs) present a promising avenue for optimizing literature review processes.

Purpose of the Study:

• To explore the application of generative artificial intelligence (GenAI) for enhancing search query formulation.
• To evaluate the performance of various massive text embedding models in document retrieval tasks.
• To compare the effectiveness of LLM-generated queries against human-crafted queries.

Main Methods:

• Utilized generative artificial intelligence (GenAI) for query reformulation.
• Evaluated nine massive text embedding models with different sizes and fine-tuning strategies.
• Applied prompt engineering techniques to assess LLM-generated queries against human-crafted queries.
• Conducted evaluations across five datasets using recall, average precision, and rank-based metrics.

Main Results:

• Text embedding models fine-tuned for semantic similarity outperformed general-purpose models.
• The UAE Large embedding model demonstrated robustness across diverse scientific domains.
• Zero-shot and few-shot prompted queries generated by LLMs frequently exceeded the performance of human-formulated queries.

Conclusions:

• Integrating LLMs and massive text embeddings can substantially reduce manual effort in literature reviews.
• GenAI serves as an effective tool for initial query formulation, with human input valuable for refinement.
• The study underscores the potential of AI to accelerate scientific discovery through improved information retrieval.