Leveraging generative AI to enhance Synthea model development.

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Computational Epidemiology

Background:

• Developing accurate agent-based patient journey models is complex and time-consuming.
• Synthetic health data generators like Synthea aim to streamline this process.
• Large language models (LLMs) offer potential for automating and enhancing model development.

Purpose of the Study:

• To explore the utility of LLMs in assisting the development of disease-specific patient journey models.
• To assess the efficiency and quality improvements offered by LLMs in conjunction with Synthea.
• To reduce the time and expertise required for creating diverse and high-quality synthetic patient data.

Main Methods:

• A four-stage methodology was employed, integrating LLMs with Synthea.
• LLMs extracted disease information, generated initial models, and refined them based on validation feedback.
• Model validation included structural/syntactic checks and requirement satisfaction assessments.

Main Results:

• Three LLMs (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) were tested using hyperthyroidism as a case study.
• Iterative refinement significantly improved model requirement fulfillment, approaching 100% for some LLMs.
• Human expert evaluation identified structural deficits in the final models despite high requirement scores.

Conclusions:

• LLMs can effectively assist in patient journey model development when integrated into structured, iterative processes with authoritative data.
• Iterative refinement is key to meeting model requirements, but human oversight is essential for clinical accuracy and structural integrity.
• This approach, while promising, requires continued human expertise for validation and addressing limitations such as medical code inaccuracies and comorbidity integration.