Development and Evaluation of a Retrieval-Augmented Generation Chatbot for Orthopedic and Trauma Surgery Patient...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Orthopedic Surgery

Background:

• Large language models (LLMs) show promise in healthcare but face challenges with factual accuracy and source traceability.
• Retrieval-augmented generation (RAG) improves LLM accuracy by integrating document retrieval.
• RAG applications in orthopedic patient education, especially in non-English contexts, are underexplored.

Purpose of the Study:

• To develop and evaluate a German-language RAG chatbot for orthopedic patient education.
• To assess the chatbot's accuracy, precision, source alignment, user satisfaction, and trustworthiness.
• To provide evidence-based information on common orthopedic conditions.

Main Methods:

• Developed a RAG chatbot using OpenAI's GPT and a Qdrant vector database.
• Utilized a corpus of 899 German orthopedic documents, including guidelines and patient information.
• Conducted human validation with 30 participants and automated evaluation using the Retrieval-Augmented Generation Assessment Scale.

Main Results:

• Human ratings showed high perceived accuracy (4.55/5), helpfulness (4.61/5), and ease of use (4.90/5).
• Automated evaluation confirmed strong technical performance in answer relevancy (0.864), contextual precision (0.891), and faithfulness (0.853).
• Performance was highest for knee and back conditions, with lower accuracy for hip-related queries.

Conclusions:

• The RAG chatbot effectively delivers orthopedic patient education in German.
• Over 9500 real-world interactions on the Orthinform platform indicate relevance and acceptance.
• Future work should enhance domain coverage, retrieval precision, and integrate advanced RAG techniques for improved safety.