Performance of Large Language Models Under Input Variability in Health Care Applications: Dataset Development and...

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Informatics

Background:

• Large language models (LLMs) are increasingly used in healthcare for patient care and decision-making.
• The reliability of LLMs with imperfect clinical data is not well understood.
• Data imperfections are common in clinical documentation and patient-generated information.

Purpose of the Study:

• Investigate the impact of input perturbations on LLM performance in health applications.
• Compare the effects of different perturbation types and levels.
• Analyze differential impacts on health-related versus non-health-related terms.

Main Methods:

• Systematic evaluation of 3 LLMs across 3 health-related tasks.
• Utilized a novel dataset with human-like variations: redactions, homophones, and typographical errors.
• Assessed performance at various perturbation levels.

Main Results:

• LLMs demonstrated notable robustness to common input variations; performance was stable or improved in over 55% of cases.
• Lower perturbation levels sometimes led to increased performance (14.07%).
• Redactions proved more detrimental to LLM performance than other variations.

Conclusions:

• Healthcare applications using LLMs must account for input variability and data quality.
• Robustness to imperfect inputs is crucial for LLM reliability in clinical settings.
• Findings offer insights for developing resilient AI tools and improving LLM performance in healthcare.