Enabling Episode-Level Transparency in Value-Based Care Through Large Language Model-Driven Provider Directories.

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare

Background:

• Conventional provider directories are critical but fragile components of the US healthcare system.
• Deficiencies in current directories, such as lack of cost/risk context and inaccurate data, limit transparency and value-based care (VBC) effectiveness.
• Episode-based payment models are hindered by inadequate provider selection tools.

Purpose of the Study:

• To evaluate a large language model (LLM)-driven provider directory chatbot for episode-based care navigation.
• To assess the performance of four LLMs (GPT-3.5-turbo, GPT-4o-mini, GPT-4o, GPT-5.1) using structured, synthetic datasets.
• To examine LLM performance in terms of output validity, episode identification, provider ranking, numeric fidelity, and hallucination risk.

Main Methods:

• Utilized 87 natural language test scenarios with strictly structured, synthetic cost and performance datasets.
• Assessed four widely used LLMs under identical deterministic conditions.
• Introduced a revised formulation of ranking correctness prioritizing accurate episode identification.

Main Results:

• All evaluated LLMs demonstrated high episode identification accuracy, approaching 91%.
• Substantial variability was observed in downstream provider ranking reliability and numeric precision among the models.
• LLM-enabled directories showed potential for enhancing transparency and user experience in VBC settings.

Conclusions:

• LLM-driven provider directories show promise for improving transparency and navigation within value-based care.
• While episode identification is strong, further optimization is needed for provider ranking accuracy and numeric fidelity before large-scale deployment.
• These preliminary findings suggest LLMs can meaningfully enhance VBC by addressing limitations of conventional provider directories.