Benchmarking large language models for de-identification of electronic health record notes

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Health Data Security

Background:

• The increasing use of large language models (LLMs) in processing clinical text necessitates robust de-identification methods.
• Evaluating LLM capabilities for identifying sensitive health information (SHI) is crucial for ensuring data privacy.
• Existing de-identification techniques require benchmarking against advanced LLM-based approaches.

Purpose of the Study:

• To conduct a comprehensive benchmarking analysis of LLM-based, rule-based, and hybrid de-identification methods.
• To evaluate the performance and robustness of various de-identification models across diverse datasets.
• To compare the effectiveness of different LLM fine-tuning strategies for SHI detection.

Main Methods:

• Utilized five diverse datasets (i2b2-2006, MIMIC-2008, i2b2-2014, i2b2-2016, OpenDeID v1) from multiple countries.
• Developed three baseline and eight LLM-based de-identification models.
• Implemented nine experimental settings to assess cross-dataset performance and model robustness.

Main Results:

• The best baseline model, trained on a combined corpus, achieved a strict F1 micro-average score of 0.8172.
• Supervised fine-tuning of LLMs using a combined corpus configuration yielded the highest performance with a strict F1 score of 0.9447.
• Performance varied significantly across heterogeneous electronic health record sources.

Conclusions:

• Harmonizing corpora is essential for standardized data formatting and SHI management, improving de-identification reliability.
• Fine-tuned LLMs demonstrate superior accuracy in de-identification but exhibit performance variability.
• Addressing inconsistencies in electronic health record data is critical for the ethical and technical deployment of LLMs in healthcare.