Enhancing antimicrobial stewardship using a retrieval-augmented generation large language model for infectious...

This study developed and evaluateed a retrieval-augmented generation (RAG)-based large language model system designed to standardize and improve antimicrobial treatment protocol adherence, leveraging the Antimicrobial Treatment Guide from the Hospital Israelita Albert Einstein in São Paulo, Brazil.

Four modules were used to compose the search system, namely document processing, query processing, large language model, and prompt management. Protocol documents were manually segmented into manageable text chunks, converted into vectors, and stored in a vector database to enable efficient retrieval using the RAG methodology. This approach allows the model to generate responses based on external data without additional training. The evaluation included quantitative metrics, such as the Jaccard index for retrieval accuracy, Public Health System (SUS - Sistema Único de Saúde) usability scale, and behavior change questionnaire to assess user perception and impact on clinical decision-making.

The system achieved an SUS score of 82, indicating excellent usability. The system further correctly retrieved the appropriate antimicrobial protocol in 83% of cases. These results suggest effective alignment between system responses and institutional clinical guidelines.

The RAG-based system demonstrated high usability and reliable retrieval performance, supporting standardized antimicrobial therapy aligned with local guidelines. This approach has the potential to bridge the gap between clinical practice and protocol adherence, enhance diagnostic accuracy and treatment specificity, and potentially reduce the risk of antibiotic resistance while improving patient outcomes.