Improving Meditron for Medical Coding Through Fine-Tuning: A Comparative Evaluation Against GPT-4

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Medical coding is crucial for healthcare billing and clinical record-keeping but presents significant challenges.
• Previous studies indicated limitations in large language model (LLM) accuracy for medical coding tasks.
• The need for efficient and accurate diagnostic coding is paramount in modern healthcare systems.

Purpose of the Study:

• To compare the performance of GPT-4 and a fine-tuned LLM (Meditron-CHUV) in generating International Classification of Diseases, 10th Revision (ICD-10) and Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT) admission diagnoses from clinical anamnesis.
• To evaluate clinician preference and performance variations across medical specialties.

Main Methods:

• A comparative study involving GPT-4 and Meditron-CHUV, fine-tuned on clinical data from Lausanne University Hospital.
• Generation of ICD-10 and SNOMED-CT diagnoses from 40 synthetic clinical vignettes based on anamnesis alone.
• Blind evaluation of generated diagnoses by five clinicians, assessing overall preference and specialty-specific performance.

Main Results:

• GPT-4 was generally preferred by clinicians, but Meditron-CHUV demonstrated superior performance in adult medicine.
• Model performance varied significantly by medical specialty (p < 0.001), but not by the clinicians' professional background.
• Despite plausible labels, SNOMED-CT codes generated by both models were systematically invalid.

Conclusions:

• Current LLMs have limitations in generating accurate and valid medical codes, particularly for SNOMED-CT.
• The choice of LLM for medical coding should consider the specific clinical specialty.
• Hybrid approaches combining LLM capabilities with retrieval-augmented generation and ontologies are recommended for improved accuracy and validity. Real-world data validation is the next step.