Reproducibility and Robustness of Large Language Models for Mobility Functional Status Extraction

Area of Science:

• Natural Language Processing
• Medical Informatics
• Artificial Intelligence in Healthcare

Background:

• Clinical narrative text is rich in patient data but challenging for information extraction (IE) due to variability.
• Large language models (LLMs) show promise for clinical IE, but their reproducibility and robustness are critical for deployment.
• Quantifying LLM stability is essential for reliable clinical applications.

Purpose of the Study:

• To evaluate the reproducibility and robustness of three open-weight large language models (LLMs) for clinical information extraction.
• To assess the impact of prompt variations and model architecture on LLM performance and stability.
• To provide recommendations for improving LLM reliability in clinical settings.

Main Methods:

• Evaluated three distinct LLMs (Llama 3.3, Llama 4, MedGemma) on binary clinical IE tasks related to the ICF mobility framework.
• Quantified intra-prompt reproducibility (repeated sampling) and inter-prompt robustness (paraphrased prompts).
• Measured predictive performance (F1-score) and stability (Fleiss' Kappa), analyzing factor effects with ANOVA.

Main Results:

• Increasing temperature generally decreased LLM agreement, with model-dependent effects.
• Prompt paraphrasing significantly reduced LLM stability, especially for Mixture-of-Experts (MoE) models.
• Self-consistency via majority voting improved stability (Kappa) and often maintained or improved performance (F1-score).

Conclusions:

• LLM reliability in clinical IE is sensitive to prompt design and model architecture.
• Self-consistency offers a practical method to enhance the stability and performance of LLMs for clinical tasks.
• A reproducible framework is presented for evaluating and improving LLM reliability in healthcare.