A clinical environment simulator for dynamic AI evaluation

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Healthcare Systems Engineering

Background:

• Current clinical large language model (LLM) evaluations use static datasets, failing to represent real-world dynamic healthcare environments.
• Existing benchmarks do not capture the cascading effects of clinical decisions on patient outcomes and hospital operations.

Purpose of the Study:

• To introduce the Clinical Environment Simulator (CES), a novel framework for evaluating clinical LLMs.
• To assess LLM performance in dynamic digital hospital environments that simulate cascading healthcare decision effects.
• To enable evaluation of temporal reasoning, resource-aware decision-making, and operational resilience in clinical LLMs.

Main Methods:

• Developed a parallel simulation architecture with a 'hospital engine' and a 'patient engine'.
• The hospital engine tracks real-time resources (beds, staff, equipment).
• The patient engine simulates disease progression and treatment responses based on LLM interventions via realistic electronic health record interfaces.

Main Results:

• The CES enables evaluation of temporal reasoning under evolving constraints.
• It facilitates assessment of resource-aware decision-making, balancing individual patient needs with system capacity.
• The framework allows for testing operational resilience through adversarial scenarios like simultaneous emergencies.

Conclusions:

• The Clinical Environment Simulator (CES) provides a more realistic and comprehensive evaluation of clinical LLMs than current benchmarks.
• CES shifts evaluation towards assessing LLMs as dynamic, integrated components of healthcare delivery.
• This framework is crucial for advancing the safe and effective deployment of AI in clinical settings.