Evaluating the Reasoning Capabilities of Large Language Models for Medical Coding and Hospital Readmission Risk...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems
• Natural Language Processing in Medicine

Background:

• Large language models (LLMs) demonstrate potential in healthcare for clinical reasoning and decision support.
• However, their reliability for critical tasks like diagnosis, coding, and risk prediction without specific training is uncertain.

Purpose of the Study:

• To evaluate and compare the zero-shot performance of reasoning and non-reasoning LLMs on clinical diagnosis, ICD-9 code prediction, and hospital readmission risk stratification.
• To assess LLMs' potential as general-purpose clinical decision support tools.

Main Methods:

• Utilized the MIMIC-IV dataset, analyzing 300 hospital discharge summaries.
• Standardized, zero-shot prompts were used across reasoning and non-reasoning LLMs, incorporating rationale elicitation for transparency.
• Performance was measured using F1-scores and correctness percentages, with statistical analysis.

Main Results:

• LLMs showed moderate success in zero-shot diagnosis and risk prediction but significantly underperformed in ICD-9 code prediction.
• OpenAI-O3 demonstrated superior performance in diagnosis and ICD-9 coding among tested models.
• Reasoning models offered marginal performance gains and improved interpretability but generated verbose outputs.

Conclusions:

• Current LLMs require task-specific fine-tuning and human-in-the-loop validation for clinical use.
• Medical coding remains a significant challenge for LLMs in zero-shot settings.
• Further research is needed to improve LLM stability, reliability, and performance on diverse clinical data.