Evaluation of six generative large language models answering pre-analytical questions in medical microbiology

Large language models (LLMs) have the potential to be integrated into diagnostics workflows, yet their reliability in pre-analytical diagnostic stewardship remains uncertain. We aimed to evaluate the performance of six LLMs answering common pre-analytical questions in the domain of medical microbiology. Next, we propose a blueprint to evaluate LLMs in medical microbiology.

We compared general-purpose models from OpenAI (GPT-3.5, GPT-4) and Meta AI (Llama2-7B, Llama2-13B), alongside two domain-enhanced versions (customised GPT-4 agent and Llama2-7B Medical) in a hybrid evaluation framework. First, we quantified the response length (word count), then assessed the similarity between LLM-generated responses and expert-reviewed reference answers (BLEU/ROUGE scores, BERTScore). Finally, thirty-three medical professionals rated the accuracy and completeness of the LLM-generated responses on a 1-7-point scale.

Responses of OpenAI models showed greater similarity to the reference set and significantly outperformed Meta AI-based models in accuracy and completeness. The customized generative pre-trained transformer (GPT) matched performance of OpenAI's general-purpose models, while generating significantly shorter responses (p<0.001). Quality ratings varied markedly between professionals, as medical microbiologists and experienced infectious disease specialists assigned significantly lower scores compared to infectious disease trainees and medical students (p<0.001).

We observed considerable variation in LLM performance and highlight differences in perceived response quality across professional groups, emphasizing the need for critical appraisal when consulting LLMs for pre-analytical guidance.