Optimizing biomedical information retrieval with a keyword frequency-driven prompt enhancement strategy.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Biomedical literature mining is complex due to interdisciplinary nature and jargon.
• Early natural language processing (NLP) struggled with nuanced language.
• Large language models (LLMs) advanced question-answering (QA) but face challenges with up-to-date, factual responses in specialized domains.

Purpose of the Study:

• To enhance prompt engineering for LLMs in biomedical QA tasks.
• To develop a retrieval-augmented architecture guiding LLMs with explicit user query signals.
• To improve the accuracy and relevance of LLM-generated responses in specialized biomedical fields.

Main Methods:

• Evaluated two prompt enhancement approaches: text embedding/vector similarity and explicit query signal extraction.
• Tested methods on 50 challenging biomedical questions against a BM25 baseline.
• Utilized GPT-4 for response generation and manual quality assessment.

Main Results:

• The explicit query signal method achieved a median Precision@10 of 0.95, significantly outperforming other approaches.
• LLM-generated responses using the proposed method received a median quality score of 2.5.
• Developed WeiseEule QA bot for comparative analysis and citation identification.

Conclusions:

• Explicit user query signals are superior to text embedding for enhancing LLM responses in specialized domains.
• This approach mitigates common LLM drawbacks like hallucinations and outdated information.
• Provides users greater control over information used by LLMs for more reliable QA.