An Extensible Evaluation Framework Applied to Clinical Text Deidentification Natural Language Processing Tools:...

Area of Science:

• Computational linguistics
• Medical informatics
• Software engineering

Background:

• Clinical natural language processing (NLP) lacks standardized evaluation frameworks for deidentification systems.
• Current methods offer limited comparability across diverse corpora and personally identifiable information (PII) categories.
• Researchers need a unified system to assess NLP performance consistently.

Purpose of the Study:

• To present an open-source, extensible framework for comparing clinical NLP deidentification system performance.
• To enable cross-corpus evaluations even with non-aligned annotation categories.
• To facilitate easier testing of new systems and corpora within a consistent evaluation pipeline.

Main Methods:

• Developed an end-to-end framework using shell scripts for extensibility.
• Evaluated six off-the-shelf deidentification systems (CliniDeID, deid, MIST, NeuroNER, NLM Scrubber, Philter).
• Tested systems across three clinical text corpora with non-analogous annotation categories, including private and public datasets.

Main Results:

• Observed substantial variations in processing speeds, with MIST being the fastest (24.57 notes/sec) and CliniDeID the slowest (1.00 notes/sec).
• No single system consistently outperformed others; performance varied across PII categories and corpora.
• Identified trade-offs between precision and recall, with CliniDeID and Philter favoring recall, and others favoring precision.

Conclusions:

• Evaluating NLP and deidentification systems in isolation limits comparative analysis.
• A unified evaluation pipeline across multiple systems and corpora enables more nuanced comparisons.
• This open pipeline aims to lower barriers for NLP system evaluation and advancement.