Benchmarking large language models for de-identification of electronic health record notes

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Health Data Privacy

Background:

• The increasing use of large language models (LLMs) in processing clinical text necessitates robust de-identification methods.
• Evaluating LLM capabilities for identifying sensitive health information (SHI) is crucial for reliable clinical text processing.

Purpose of the Study:

• To benchmark LLM-based, rule-based, and hybrid de-identification methods.
• To assess the performance and robustness of various de-identification approaches across diverse datasets.

Main Methods:

• Utilized five international datasets (i2b2-2006, MIMIC-2008, i2b2-2014, i2b2-2016, OpenDeID v1).
• Developed three baseline and eight LLM-based models.
• Implemented nine experimental settings to evaluate cross-dataset performance and model robustness.

Main Results:

• The best baseline model achieved a strict F1 micro-average score of 0.8172 when trained on a combined corpus.
• Supervised fine-tuning of LLMs (setting 9) yielded the highest performance with a strict F1 score of 0.9447.
• Harmonizing corpora improved data standardization and SHI management.

Conclusions:

• Fine-tuned LLMs demonstrate superior accuracy in de-identification.
• Performance variability across diverse electronic health record sources presents technical challenges for LLM implementation.
• Addressing data inconsistencies is vital for the ethical and technical deployment of LLMs in handling sensitive health data.