Benchmarking transformer-based models for medical record de-identification in a single center multi-specialty...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Data Privacy

Background:

• Patient confidentiality is essential for electronic health record (EHR) research.
• Manual redaction of sensitive information is time-consuming and not scalable.
• Automated de-identification methods offer a promising alternative for data sharing.

Purpose of the Study:

• To evaluate the performance of various automated de-identification models on clinical records.
• To compare task-specific transformer models against large language models (LLMs).
• To assess the accuracy and adaptability of different de-identification approaches.

Main Methods:

• Evaluated four transformer-based models and five LLMs on 3,650 UK clinical records.
• Utilized dual-annotation by clinicians for precise performance comparison.
• Assessed models across general and specialty datasets.

Main Results:

• Microsoft Azure de-identification service achieved the highest F1 score, comparable to clinician performance.
• Fine-tuned AnonCAT and GPT-4-0125 (few-shot) also demonstrated strong results.
• Smaller LLMs exhibited issues with over-redaction and hallucination; task-specific models showed better dataset stability.

Conclusions:

• Automated de-identification systems can effectively support large-scale clinical record sharing.
• Model selection, adaptation strategies, and privacy safeguards are critical for successful implementation.
• These systems enhance data utility while ensuring robust patient privacy.