Evaluation of a Retrieval-Augmented Generation Chatbot for Antimicrobial Resistance Research: Comparative Analysis of...

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Medicine
• Computational Biology

Background:

• Antimicrobial resistance (AMR) is a significant global health challenge, impacting antibiotic efficacy and clinical decisions.
• Synthesizing extensive AMR literature is time-consuming for researchers and clinicians.
• Large language models (LLMs) present opportunities to improve access to specialized biomedical knowledge.

Purpose of the Study:

• Develop a retrieval-augmented generation (RAG) chatbot for analyzing antimicrobial resistance (AMR) literature.
• Compare the performance, cost, and scalability of various commercial and open-source LLMs for AMR research.

Main Methods:

• Compiled a corpus of 164 AMR articles and embedded them into a ChromaDB vector database.
• Implemented a RAG chatbot utilizing five LLM backbones: GPT-4, GPT-4o, GPT-4o-mini, Claude 3.7 Sonnet, and LLaMA 4 Maverick.
• Evaluated models based on correctness, faithfulness, relevancy, computational cost, and latency using a synthetic dataset.

Main Results:

• All tested LLMs produced scientifically accurate responses within the RAG framework.
• GPT-4o achieved high accuracy at a significantly lower cost and faster response time compared to GPT-4.
• LLaMA 4 Maverick and GPT-4o-mini provided cost savings with reduced accuracy, while Claude 3.7 Sonnet had a less favorable cost-performance ratio.

Conclusions:

• RAG-based chatbots can enhance AMR research by providing accurate and scalable access to scientific literature.
• The study highlights the performance-cost-speed trade-offs among different LLMs, aiding selection for clinical and research applications.
• Future work will explore language-specific embeddings and domain agents to further improve chatbot accuracy and clinical utility.