Magnitude and Impact of Hallucinations in Tabular Synthetic Health Data on Prognostic Machine Learning Models:...

Area of Science:

• Health Informatics
• Artificial Intelligence
• Machine Learning

Background:

• Generative AI for synthetic data generation (SDG) can advance health research.
• Hallucinations, a known issue in text AI, can also occur in tabular SDG.
• Understanding these hallucinations is crucial for reliable health data innovation.

Purpose of the Study:

• Investigate the frequency of hallucinations in tabular synthetic health data.
• Determine if hallucination frequency correlates with training data complexity.
• Assess the impact of hallucinations on the utility of synthetic data for prognostic machine learning (ML) models.

Main Methods:

• Generated 6354 synthetic datasets from 12 real-world health datasets of varying complexity.
• Utilized 7 different SDG models to create synthetic data.
• Defined hallucinations as non-population records and measured hallucination rate (HR).
• Evaluated downstream prognostic performance using ML (light gradient boosted machine) and artificial neural networks (multilayer perceptron).

Main Results:

• Hallucination rates varied widely (0.3%–100%), with a median of 99.1%, and increased with data complexity.
• Despite high hallucination rates, most SDG models showed no significant impact on ML prognostic model performance.
• Observed minimal performance decreases (max AUC change of -0.0002) in models where an association was detected.

Conclusions:

• Hallucinations are prevalent in synthetic tabular health data.
• High hallucination rates do not necessarily compromise the utility of synthetic data for prognostic modeling.
• Synthetic health data remains a valuable resource for developing predictive ML models.