Retrieval-augmented generation for interpreting clinical laboratory regulations using large language models.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Regulatory Science

Background:

• Large language models (LLMs) excel at general knowledge but struggle with accuracy in specialized domains.
• Retrieval-augmented generation (RAG) improves LLM accuracy by grounding outputs in specific source documents.
• Accuracy and consistency are paramount for question-answering systems in critical fields like laboratory medicine.

Purpose of the Study:

• To develop and evaluate Raven, a custom RAG system for answering laboratory regulatory questions.
• To assess Raven's accuracy and reliability using the Code of Federal Regulations (CFR) as an authoritative source.
• To determine the potential of RAG systems as decision-support tools in healthcare.

Main Methods:

• Developed Raven, a RAG system integrating a vector search pipeline and an LLM.
• Utilized 42 CFR Part 493 (laboratory regulations) as the knowledge base for Raven.
• Tested Raven with 103 synthetic regulatory questions, comparing its responses to those of a board-certified pathologist.

Main Results:

• Raven achieved 92.0% complete and correct answers for questions explicitly addressed in the CFR.
• Suboptimal responses were primarily due to retrieval issues, not LLM hallucination.
• Performance decreased for questions outside the CFR scope, validating the system's grounding.

Conclusions:

• Basic RAG systems can provide accurate, verifiable answers to complex regulatory questions.
• Raven demonstrated utility as a decision-support system for laboratory regulatory inquiries.
• RAG tools, with proper integration, hold promise for knowledge-intensive healthcare domains.