Bootstrapping a de-identification system for narrative patient records: cost-performance tradeoffs

Area of Science:

• Medical Informatics
• Natural Language Processing
• Health Data Security

Background:

• Clinical records contain sensitive Protected Health Information (PHI) requiring robust de-identification.
• Existing de-identification systems often require significant development effort and resources.
• Open-source toolkits offer a potential solution for developing custom de-identification systems.

Purpose of the Study:

• To develop and evaluate a de-identification system for clinical records using the open-source MITRE Identification Scrubber Toolkit (MIST).
• To quantify the human annotation effort required to achieve high accuracy in de-identification.
• To assess the performance of the developed system against established benchmarks.

Main Methods:

• Iterative development of statistical de-identification models using MIST on history and physical notes, and social work notes.
• Annotation of clinical records in rounds, followed by model training, application, and correction.
• Performance measurement (precision, recall, F-score) and time-to-annotation tracking at each stage.

Main Results:

• An F-score of 0.89 was achieved after 33 minutes of annotation, with an F-score of 0.95 reached after approximately 8 hours.
• Annotation effort and time decreased as model performance improved through iterative training.
• History and physical notes showed higher de-identification accuracy than social work notes due to PHI variability.

Conclusions:

• A functional de-identification system can be developed from scratch using the MIST framework with a reasonable amount of human effort.
• The developed MIST-based system demonstrates performance comparable to other leading de-identification solutions.
• The study highlights the feasibility and efficiency of using open-source toolkits for building accurate clinical record de-identification systems.