Evaluating the Performance of LLMs in ICF Classification: Insights from Medical and General Models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Health Classification Systems

Background:

• Unstructured clinical text data, including patient anecdotes and insights, is often underutilized in healthcare.
• Large Language Models (LLMs) offer potential for processing and structuring this complex medical data.
• The World Health Organization's International Classification of Functioning, Disability, and Health (ICF) provides a standardized framework for describing health and disability.

Purpose of the Study:

• To investigate the application of medically fine-tuned LLMs for automated ICF classification.
• To compare the efficiency of medical LLMs (MedAlpaca, Meditron) against general-purpose LLMs (ChatGPT, Claude) in processing real medical cases.
• To evaluate the performance of specialized vs. general LLMs in the context of rehabilitation and intensive care unit data.

Main Methods:

• Utilized medically fine-tuned LLMs (MedAlpaca, Meditron) and general-purpose LLMs (ChatGPT, Claude).
• Applied LLMs to real-world medical case data from rehabilitation and intensive care settings.
• Benchmarked the performance of different LLMs for automated ICF code generation.

Main Results:

• Medical LLMs demonstrate potential for ICF classification tasks.
• General-purpose LLMs performed comparably to specialized medical LLMs.
• The complexity of ICF classification may require deeper contextual understanding than current models provide.

Conclusions:

• Automated ICF classification using LLMs is feasible but requires careful model selection.
• Specialized medical LLMs do not inherently outperform general LLMs for complex ICF coding.
• Further research is needed to enhance LLM contextual understanding for improved medical classification accuracy.