Evidence-Grounded LLM Validation of MIMIC-IV ICD Labels.

Area of Science:

• Clinical Natural Language Processing (NLP)
• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Automated International Classification of Diseases (ICD-10) code assignment from discharge summaries is crucial for clinical NLP.
• Existing benchmarks like MIMIC-IV suffer from label noise, with many codes lacking textual evidence or being mis-specified.

Purpose of the Study:

• To develop and evaluate a large language model (LLM)-based workflow for validating ICD-10 codes against clinical notes.
• To create refined datasets by verifying code-evidence support and suggesting evidence-based replacements.
• To assess the impact of data refinement on the performance of state-of-the-art ICD coding models.

Main Methods:

• An LLM-based evidence-validation workflow was designed to assess (note, code) pairs.
• The workflow identifies text-supported codes, extracts evidence, and suggests code replacements.
• The pipeline was applied to 10,000 MIMIC-IV notes, generating Evidence-Verified (EV) and Evidence-Replaced (ER) label sets.
• Six ICD coding models were replicated and evaluated using paired bootstrap resampling for micro-precision, recall, and F1 scores.

Main Results:

• Removing unsupported charted codes from MIMIC-IV significantly enhanced model performance.
• The refined label sets (EV and ER) led to more trustworthy benchmarks for automated medical coding.
• Models evaluated on the refined datasets demonstrated improved micro-precision, recall, and F1 scores.

Conclusions:

• LLM-based evidence validation is effective in reducing label noise in clinical NLP benchmarks.
• Refined datasets improve the reliability and performance of automated ICD-10 coding systems.
• This approach offers a pathway to more accurate and trustworthy automated medical coding.