Transferability of neural network clinical deidentification systems

Area of Science:

• Natural Language Processing (NLP)
• Machine Learning in Healthcare
• Clinical Data Deidentification

Background:

• Current neural network deidentification studies often require large, in-house annotated datasets.
• Real-world scenarios frequently involve limited or no in-house training data for deidentification.
• The optimal utilization of existing systems and external data for deidentification remains unclear.

Purpose of the Study:

• To investigate the transferability of the NeuroNER neural clinical deidentification system across diverse datasets.
• To evaluate architectural modifications for domain generalization and strategic training for domain transfer.
• To determine the most effective methods for utilizing external data in clinical deidentification tasks.

Main Methods:

• Comparative study of NeuroNER transferability across 4 clinical note corpora from 2 institutions.
• Architectural modification of NeuroNER with 2 domain generalization approaches.
• Evaluation using 3 distinct training strategies to assess transferability across sources, note types, and institutions.

Main Results:

• Transferability from a single external source yielded inconsistent results.
• Utilizing multiple external sources consistently achieved an F1-score of approximately 80%.
• Fine-tuning proved to be a dominant and effective transfer strategy, outperforming other methods.

Conclusions:

• External data sources are valuable for clinical deidentification, even when limited in-house data exists.
• Fine-tuning strategies, with or without domain generalization, are crucial for successful knowledge transfer.
• Cross-institutional transferability improved performance but varied by note type and annotation label.