LLM-assisted clinical coding audit through an interpretable coding pipeline

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Data Science

Background:

• Clinical coding is essential but complex, often facing challenges with imperfect training data.
• Undercoding and coding errors in standard datasets are overlooked issues impacting automated coding algorithms.
• Auditing clinical coding data is a difficult but necessary process.

Purpose of the Study:

• To address undercoding and coding errors in clinical datasets.
• To investigate the impact of data quality issues on automated coding algorithms.
• To explore AI-assisted tools for clinical coding audits.

Main Methods:

• Developed an interpretable coding pipeline integrating Large Language Models (LLMs) for evidence extraction and code verification.
• Utilized a multiclass classifier trained on a large-scale silver-standard evidence dataset for code prediction.
• Employed the pipeline as an audit tool for professional coders to identify and correct errors in MDACE and CodiEsp datasets.

Main Results:

• Identified significant data quality issues: 76.3% undercoding in MDACE and 29.7% error rate in CodiEsp.
• Re-evaluating models on corrected datasets improved performance across metrics.
• The proposed pipeline demonstrated superior or comparable performance to state-of-the-art LLM-based methods.

Conclusions:

• Substantial errors in benchmark datasets significantly impact clinical AI model evaluation.
• There is a critical need to shift research focus towards data-centric solutions in clinical AI.
• AI-assisted audits are effective in improving clinical data integrity.