Evaluating the Performance of LLMs in ICF Classification: Insights from Medical and General Models

In the medical field, text data comprising personal anecdotes and detailed patient insights are often underutilized due to their unstructured nature and variability among clinicians. However, recent advances in Large Language Models (LLMs) present an opportunity to harness this data effectively. This paper explores the use of the International Classification of Functioning, Disability, and Health (ICF) framework recommended by the World Health Organization (WHO), which offers a holistic approach considering personal and environmental factors along with impairments, to structure textual descriptions systematically. The study investigates the application of medically fine-tuned LLMs, such as MedAlpaca and Meditron, for automated ICF creation, comparing their efficiency in processing real medical cases from two distinct contexts: rehabilitation and intensive care units. Additionally, we benchmark medical LLMs against general-purpose LLMs, including ChatGPT and Claude, to assess whether specialized models truly offer an advantage in medical classification tasks. Preliminary findings indicate that while medical LLMs show potential for ICF classification tasks, they may not necessarily outperform general-purpose models, as the complexity of ICF requires a deeper level of contextual understanding.International classification of functioning, disability and health (ICF), Intensive care, large language models, AI, medical, LLM, Rehabilitation.