Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and...

Area of Science:

• Natural Language Processing
• Medical Informatics
• Computational Linguistics

Background:

• Named Entity Recognition (NER) is crucial for extracting medical information from health records.
• Developing NER for low-resource languages like Estonian is challenging due to limited annotated data.
• Large Language Models (LLMs) show promise for text understanding across languages and domains.

Purpose of the Study:

• To develop medical NER models for Estonian, a low-resource language.
• To generate synthetic Estonian health data using LLMs for NER model training.
• To preserve patient data privacy by avoiding real-world annotated data.

Main Methods:

• A three-step pipeline: synthetic data generation (GPT-2), LLM annotation (GPT-3.5-Turbo, GPT-4), and NER model fine-tuning.
• Comparison of different LLM prompts and models (GPT-3.5-Turbo, GPT-4, local LLM).
• Exploration of the impact of synthetic data volume on NER model performance.

Main Results:

• The methodology shows potential for extracting medical entities from real-world texts.
• The best setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction.
• Performance varies across entity types, with procedures being more complex to extract.

Conclusions:

• LLMs can be effectively leveraged with synthetic data to train NER models, preserving patient privacy.
• This approach offers a promising solution for developing NER models in low-resource languages like Estonian.
• Future work will focus on refining synthetic data generation and expanding applicability to other domains and languages.