Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• The World Health Organization's International Classification of Diseases (ICD) standardizes health condition coding for policy, research, and billing.
• Current AI automation in medical coding shows promise but lags human accuracy and lacks explainability for clinical adoption.

Purpose of the Study:

• To explore the potential of large language models (LLMs) for assisting medical coders with ICD-10 coding.
• To develop a computer-assisted coding system augmenting human coders by identifying lead terms and using retrieval-augmented generation (RAG).

Main Methods:

• Utilized the CodiEsp-X dataset (1000 Spanish clinical cases with ICD-10 codes) and created CodiEsp-X-lead by replacing full-text evidence with lead terms using GPT-4.
• Fine-tuned a Robustly Optimized BERT Pretraining Approach (ROBERTa) model for named entity recognition to extract lead terms.
• Employed GPT-4 for generating code descriptions and a RAG approach with OpenAI's text-embedding-ada-002 to assign ICD codes to lead terms via a vector database.

Main Results:

• The fine-tuned ROBERTa model achieved an F1-score of 0.80 for ICD lead term extraction on the CodiEsp-X-lead dataset.
• GPT-4-generated code descriptions reduced RAG retrieval failures by approximately 5% for diagnoses and procedures.
• The overall explainability F1-score for the CodiEsp-X task was 0.305, significantly below the state-of-the-art (0.633), due to reliance on descriptions and workflow misalignment.

Conclusions:

• Lead term extraction shows potential, but RAG-based code assignment using GPT-4 and descriptions proved less effective.
• Future research must better integrate AI with medical coder workflows, including official coding guidelines and alphabetic indexes, to enhance accuracy and practical utility.