Evaluation and Bias Analysis of Large Language Models in Generating Synthetic Electronic Health Records: Comparative...

Area of Science:

• Artificial Intelligence in Healthcare
• Medical Informatics
• Health Equity

Background:

• Synthetic electronic health records (EHRs) generated by large language models (LLMs) offer privacy-preserving solutions for clinical education and model training.
• However, underexplored performance variations and demographic biases in LLM-generated EHRs pose risks to equitable healthcare.

Purpose of the Study:

• To systematically assess the performance of various LLMs in generating synthetic EHRs.
• To critically evaluate gender and racial biases in LLM-generated EHRs across 20 diseases with varying demographic prevalence.

Main Methods:

• Developed a framework to generate 140,000 synthetic EHRs using 7 LLMs and 10 prompts.
• Introduced the electronic health record performance score (EPS) for completeness and statistical parity difference (SPD) for demographic bias assessment.
• Utilized chi-square tests to evaluate bias across demographic groups.

Main Results:

• Larger LLMs demonstrated superior EHR generation performance (higher EPS) but exhibited increased gender and racial biases.
• Observed sex polarization, with amplified representation for female-dominated diseases and skewed male representation in others.
• Identified racial biases, including overestimation of White/Black populations and underestimation of Hispanic/Asian groups in synthetic EHRs.

Conclusions:

• A performance-bias trade-off exists: larger LLMs generate more comprehensive EHRs but with heightened demographic biases.
• Biases were present across all tested LLMs, not exclusively larger models.
• Findings underscore the urgent need for bias mitigation strategies and fairness benchmarks in healthcare AI.