Constructing synthetic datasets with generative artificial intelligence to train large language models to classify...

Area of Science:

• Medical informatics
• Artificial intelligence in healthcare
• Natural language processing

Background:

• Clinical notes are vital for training AI models in healthcare.
• Access to authentic clinical notes is often limited due to privacy concerns.
• Synthetic data generation offers a potential solution to data scarcity.

Purpose of the Study:

• To evaluate the performance of a language model classifier trained on synthetic clinical notes compared to authentic notes for identifying acute renal failure.
• To determine if synthetic data can achieve comparable results to real-world data in medical AI applications.

Main Methods:

• A classifier utilizing language models was developed to detect acute renal failure.
• Four training datasets were compared: authentic clinical notes (MIMIC-III) and three sets of synthetic notes generated by ChatGPT (15, 30, and 45 sentences).
• Performance was assessed using the area under the receiver operating characteristics curve (AUC) on a MIMIC-III test set.

Main Results:

• The RoBERTa model achieved an AUC of 0.84 when trained on authentic MIMIC-III notes.
• Training with synthetic notes generated by ChatGPT resulted in AUCs of 0.80 (GPT-15), 0.84 (GPT-30), and 0.76 (GPT-45).
• Performance with synthetic data closely matched that of authentic data, particularly with the GPT-30 set.

Conclusions:

• Language models trained on synthetic clinical notes demonstrate comparable performance to those trained on authentic notes for acute renal failure detection.
• The findings suggest that synthetic data can be a viable alternative to authentic clinical notes, potentially reducing the need for protected health information in AI model training.
• This research supports the use of generative AI for creating training datasets, paving the way for more accessible and privacy-preserving AI development in medicine.