BioACE: An Automated Framework for Biomedical Answer and Citation Evaluations

Area of Science:

• Biomedical Natural Language Processing
• Artificial Intelligence in Healthcare
• Scientific Literature Evaluation

Background:

• Large language models (LLMs) are increasingly used for biomedical question answering, necessitating robust evaluation methods.
• Evaluating LLM-generated text in the biomedical domain is complex due to specialized terminology and the need for expert scientific assessment.
• Existing methods struggle with the nuances of biomedical text generation, including question answering and retrieval-augmented generation (RAG).

Purpose of the Study:

• To introduce BioACE, an automated framework for evaluating biomedical answers and their supporting citations generated by LLMs.
• To develop and validate automated metrics for assessing answer completeness, correctness, precision, and recall against ground-truth facts.
• To compare the effectiveness of various approaches, including natural language inference (NLI) and other language models, for citation quality evaluation.

Main Methods:

• Developed BioACE, an automated framework for biomedical answer and citation evaluation.
• Implemented automated approaches to assess answer completeness, correctness, precision, and recall.
• Utilized natural language inference (NLI), pre-trained language models, and LLMs to evaluate citation quality and evidence.
• Conducted extensive experiments to correlate automated evaluations with human assessments.

Main Results:

• The BioACE framework provides automated evaluation of biomedical answers and citations.
• Automated metrics demonstrated correlation with human evaluations for answer quality assessment.
• Analysis identified optimal approaches for evaluating biomedical answer and citation quality.
• The study offers a comprehensive evaluation package for LLM-generated biomedical content.

Conclusions:

• BioACE presents a significant advancement in the automated evaluation of LLM-generated biomedical text.
• The framework addresses the critical need for reliable assessment of answers and citations in the biomedical domain.
• BioACE facilitates the responsible development and deployment of LLMs for biomedical applications.
• The findings guide the selection of best practices for biomedical answer and citation evaluation.