Evaluating real-world deployment of an HL7-CDA-aligned LLM for ICD-10-CM coding.

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Health Information Management

Background:

• Accurate ICD-10-CM coding is crucial for hospital operations but presents significant challenges.
• The real-world effectiveness of AI for medical coding is not well-established.
• Existing AI solutions often lack integration into diverse clinical documentation environments.

Purpose of the Study:

• To develop and evaluate a scalable AI pipeline for ICD-10-CM coding.
• To assess the real-world performance and impact of AI-assisted coding in a clinical setting.
• To identify factors influencing the successful adoption of AI in medical coding workflows.

Main Methods:

• Developed a modular AI pipeline using principled base-model selection and redundancy-aware training.
• Employed LLM-as-judge evaluation and Plackett-Luce ranking to identify high-performing foundation models (BioMistral).
• Conducted a 13-week human-in-the-loop randomized controlled trial with certified coding specialists.

Main Results:

• BioMistral demonstrated consistent performance across two institutions.
• AI-assisted workflows significantly reduced coding time while maintaining accuracy.
• User satisfaction varied based on experience, certification, and generational factors, highlighting human element importance.

Conclusions:

• Methodologically grounded AI systems can achieve robust and operationally meaningful performance in clinical documentation.
• Successful AI adoption in medical coding depends on multiple levels: infrastructure, workflow, and user acceptance.
• Model accuracy alone is insufficient for ensuring real-world impact; human factors are critical.