Extracting International Classification of Diseases Codes from Clinical Documentation Using Large Language Models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Large language models (LLMs) demonstrate potential in professional domains like medicine and law.
• Their efficacy in specialized medical tasks, such as ICD-10-CM code extraction, is not well-established.

Purpose of the Study:

• To evaluate and compare the performance of various LLMs in extracting ICD-10-CM codes against human coders.
• To identify specific challenges and reasons for discrepancies in LLM-based code extraction.

Main Methods:

• Six LLMs (GPT-3.5, GPT-4, Claude 2.1, Claude 3, Gemini Advanced, Llama 2-70b) were assessed.
• Deidentified inpatient notes from the AHIMA Vlab were used for evaluation.
• Percent agreement and Cohen's kappa were calculated; discrepancies were analyzed in a subset.

Main Results:

• LLMs extracted more unique ICD-10-CM codes than human coders, but agreement was low (kappa values: -0.02 to 0.01).
• GPT-4 showed the highest percent agreement (15.2%) for all codes; Claude 3 led for primary diagnosis (26% agreement, 0.25 kappa).
• Discrepancies included unconfirmed diagnoses, nonspecific codes, symptom coding, and hallucinations.

Conclusions:

• Current LLMs exhibit poor performance in extracting ICD-10-CM codes from inpatient notes.
• Significant improvements are needed for reliable clinical application of LLMs in medical coding.