A strategy for cost-effective large language model use at health system-scale

Area of Science:

• Artificial Intelligence in Medicine
• Health Informatics
• Computational Health

Background:

• Large language models (LLMs) show promise for optimizing clinical workflows.
• However, the economic and computational demands of implementing LLMs at a health system scale remain largely unexamined.
• Understanding these limitations is crucial for effective enterprise-level adoption.

Purpose of the Study:

• To evaluate the impact of concatenating multiple clinical notes and tasks on LLM performance under varying computational loads.
• To assess the accuracy and output formatting capabilities of different LLMs.
• To identify the limits of LLM utilization and explore cost-efficiency strategies.

Main Methods:

• Assessed ten LLMs of varying capacities and sizes using real-world patient data.
• Conducted over 300,000 experiments with diverse task sizes and configurations.
• Measured question-answering accuracy and output formatting reliability.

Main Results:

• Model performance degraded as the number of questions and notes increased.
• High-capacity models (Llama-3-70b, GPT-4-turbo-128k) demonstrated resilience, maintaining high accuracy.
• GPT-4-turbo-128k performance declined after 50 tasks with large prompts.
• Concatenation of up to 50 simultaneous tasks proved effective for these models.
• Economic analysis revealed up to a 17-fold cost reduction at 50 tasks.

Conclusions:

• LLMs can effectively concatenate up to 50 simultaneous tasks, particularly high-capacity models, after addressing minor failures.
• This approach significantly reduces costs for enterprise-scale healthcare applications.
• The study delineates LLM utilization limits and highlights pathways for cost-effective deployment in healthcare.