BioACE: An automated framework for biomedical answer and citation evaluations

Large Language Models (LLMs) are increasingly used to generate answers to medical questions; however, automated evaluation of the completeness and correctness of the answers remains a challenge, which presents a bottleneck for the development and deployment of question answering systems that are needed to address medical information needs of patients and clinicians. Our goal is to develop an automated evaluation framework for rapid testing of different question answering approaches.

We propose BioACE, an automated framework for the evaluation of completeness and correctness of medical answers and the evidence provided to support the facts stated in the answers. We compare BioACE to multiple existing approaches that evaluate the quality of an answer and the evidence provided to support the generated answers, such as natural language inference (NLI) and pre-trained and large language models (PLMs and LLMs).

The Llama-3.3-70B-Instruct (zero-shot) model achieved the highest correlation with human judgment in evaluating different aspects of completeness of the answers. Among the evaluated PLM- and LLM-based models, RoBERTa_Large achieved the best overall performance for answer correctness, while BERT_Large also performed competitively and obtained the best results in fully automated classification settings.

The Llama-3.3-70B-Instruct model can be used to evaluate answer completeness during system development. BERT_Large can be used for answer-correctness evaluation without requiring human document-relevance assessment. While our approach to the evaluation of the evidence provided to support the answers outperforms the existing approaches, it still leaves ample room for improvement, demonstrating the complexity of fact attribution. The BioACE (https://github.com/deepaknlp/BioACE/) framework provides a means for rapid evaluation of medical answers, as well as reliable ranking of question answering systems during development. The existing automated evaluation approaches do not yet allow for system deployment without human review.