An evaluation of existing text de-identification tools for use with patient progress notes from Australian general...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Health Data Security

Background:

• Digitized patient progress notes are valuable for research but require de-identification.
• Existing automated de-identification tools may not suit diverse clinical documentation practices.
• Customization is necessary for effective and ethical use of general practice notes.

Purpose of the Study:

• Evaluate the performance of four de-identification tools.
• Assess their suitability for Australian general practice progress notes.
• Identify the most promising tool for customization.

Main Methods:

• Compared three rule-based (HMS Scrubber, MIT De-id, Philter) and one machine learning (MIST) tool.
• Manually annotated 300 patient progress notes for personally identifying information.
• Measured recall, precision, f1-score, and f2-score; conducted error analysis.

Main Results:

• Philter achieved the highest aggregate recall (67%) and NAME recall (87%).
• HMS Scrubber had the highest DATE recall (94%); all tools struggled with LOCATION.
• MIST showed high precision for NAME and DATE; Philter required rule adjustments for precision.

Conclusions:

• Off-the-shelf de-identification tools need modification for Australian general practice notes.
• Philter is a promising candidate due to high recall and flexibility.
• Extensive revision of Philter's rules and dictionaries is necessary for optimal performance.