Can GPT-3.5 generate and code discharge summaries?

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Natural Language Processing

Background:

• Accurate medical coding is crucial for healthcare data analysis and reimbursement.
• Low-resource labels in medical datasets hinder the training of effective coding models.
• Large language models (LLMs) show potential for automating medical document processing.

Purpose of the Study:

• To investigate the utility of GPT-3.5 for generating and coding medical documents with International Classification of Diseases (ICD)-10 codes.
• To assess the effectiveness of GPT-3.5-driven data augmentation for improving neural coding models, particularly for low-resource labels.
• To evaluate the clinical acceptability and coding accuracy of GPT-3.5-generated medical documents.

Main Methods:

• Generated 9606 discharge summaries using GPT-3.5 based on ICD-10 code descriptions from the MIMIC-IV dataset.
• Created an augmented training set by combining generated data with the baseline training set.
• Trained and evaluated neural coding models on both baseline and augmented data, reporting micro- and macro-F1 scores.
• Utilized Weak Hierarchical Confusion Matrices to analyze coding errors.
• Assessed GPT-3.5's coding performance on self-generated and real MIMIC-IV data.
• Conducted clinical evaluations of generated documents.

Main Results:

• Data augmentation led to slightly decreased overall model performance but enhanced performance for rare (generation) ICD-10 codes and their families.
• Augmented models demonstrated reduced out-of-family coding errors.
• GPT-3.5 accurately identified ICD-10 codes based on prompted descriptions but underperformed on real-world medical data.
• Clinical evaluators found generated concepts to be correct but noted deficiencies in narrative variety and supporting information.

Conclusions:

• GPT-3.5, within the tested prompt setting, is not suitable for standalone ICD-10 coding but can support data augmentation for training neural models.
• Data augmentation positively impacts generation code families, especially those with existing examples, and reduces out-of-family errors.
• Generated documents correctly reflect prompted concepts but lack narrative authenticity and variety, requiring further refinement for clinical use.