Synthetic Patient-Physician Conversations Simulated by Large Language Models: A Multi-Dimensional Evaluation

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Medical Education Technology

Background:

• Healthcare AI faces data accessibility challenges due to privacy and logistics.
• Synthetic data offers a solution by mimicking real patient information without compromising privacy.
• Large Language Models (LLMs) present new opportunities for generating realistic clinical conversations.

Purpose of the Study:

• To evaluate the performance of four leading LLMs in generating synthetic patient-physician interaction transcripts.
• To assess the realism, accuracy, and practical relevance of LLM-generated clinical dialogues in plastic surgery scenarios.

Main Methods:

• Four LLMs (ChatGPT 4.5, ChatGPT 4o, Claude 3.7 Sonnet, Gemini Pro 2.5) generated transcripts for 10 plastic surgery scenarios each.
• Clinically trained raters evaluated transcripts using a 7-criterion rubric (Medical Accuracy, Realism, Persona Consistency, Fidelity, Empathy, Relevancy, Usability) on a 5-point Likert scale.
• Automated linguistic and content-based metrics were also employed for analysis.

Main Results:

• All LLMs demonstrated strong performance, with average ratings above 4.5 across all criteria.
• Gemini 2.5 Pro excelled in Medical Accuracy, Realism, Persona Consistency, Relevancy, and Usability.
• Claude 3.7 Sonnet led in Empathy, while ChatGPT 4.5 showed high Empathy and Usability scores. No statistically significant differences were found between models.
• Automated analysis revealed variations in dialogue length and emotional expressiveness, with Gemini 2.5 Pro generating the longest and most expressive dialogues.

Conclusions:

• Leading LLMs can produce medically accurate and emotionally appropriate synthetic dialogues for medical education and research.
• Demographic homogeneity in generated patients necessitates improvements in diversity and bias mitigation.
• LLM-generated dialogues can be cautiously integrated into medical training, simulation, and research contexts.