AI-driven approach for creating and evaluating a synthetic dataset for Medication Errors

This study aims to create a complete Medication Error (ME) dataset. This will help to address the challenge of limited access to real-world data for developing machine learning models in healthcare applications.

We use transformer-based models (GPT-4, LLAMA3, and Mistral) to create our synthetic dataset in French. These models generate a diverse range of descriptions that capture the variability of ME types. We assess the effectiveness of our synthetic dataset through expert evaluations by healthcare professionals and an AI-driven analysis, to test its realism and its utility in training machine learning models for ME classification.

The synthetic dataset demonstrates high accuracy and realism in representing diverse ME scenarios. Expert evaluation confirms that the dataset is similar to real-world ME data. The AI-driven evaluation also shows that models trained on synthetic data achieved robust classification performance, validating the dataset's utility for the development of effective ME classification tools.

The proposed approach demonstrates the potential of large language models to generate realistic synthetic ME reports in French. Out of 200 evaluated reports, 70% of zero-shot outputs were deemed below expectations, while 80% of one-shot and few-shot outputs were considered valid or valid with minor revisions by clinical experts. Furthermore, classifiers trained on 800 synthetic reports attained an F1-score of up to 0.78 when tested on real data. These results confirm that synthetic data can effectively support AI-driven ME analysis in contexts where real-world data is limited or unavailable.