Assessing Retrieval-Augmented Large Language Model Performance in Emergency Department ICD-10-CM Coding Compared to...

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Clinical Documentation Improvement

Background:

• Accurate medical coding is crucial for healthcare operations but is often complex, time-consuming, and prone to bias.
• This study addresses the challenges in medical coding by comparing AI-driven methods with traditional provider assignments.

Purpose of the Study:

• To evaluate the performance of Retrieval-Augmented Generation (RAG)-enhanced Large Language Models (LLMs) in generating ICD-10-CM codes from emergency department (ED) clinical records.
• To compare the accuracy and specificity of AI-generated codes against those assigned by healthcare providers.

Main Methods:

• A retrospective cohort study analyzed 500 ED visits, integrating data from over 1 million past visits into LLMs via RAG.
• Nine commercial and open-source LLMs were assessed, with their generated ICD-10-CM codes compared to provider-assigned codes.
• A panel of physicians and LLMs conducted a blinded review to assess the accuracy and specificity of the codes.

Main Results:

• RAG-enhanced LLMs demonstrated superior accuracy and specificity compared to provider coders.
• In cases of discrepancy, human reviewers favored GPT-4's accuracy and specificity significantly over provider codes (p<0.001).
• Even smaller open-access models like Llama-3.1-70B showed substantial improvement with RAG, and exact match rates increased significantly across all models post-RAG integration.

Conclusions:

• RAG-enhanced LLMs offer a significant improvement in medical coding accuracy and specificity within ED settings.
• The findings suggest that generative AI, particularly RAG-enhanced LLMs, has strong potential for integration into clinical workflows.
• This technology can enhance clinical outcomes and substantially reduce the administrative burden in healthcare.