Related Concept Videos
Deindividuation
PPE Use in Healthcare Settings II: Doffing
Ethical Standards II
Nurses are entrusted with upholding various ethical principles and standards. Nurses forge solid therapeutic relationships using trust, empathy, autonomy, confidentiality, and professional competence.
Confidentiality is crucial, embodying respect for individual privacy...
MALDI-TOF Mass Spectrometry
Matrix-assisted laser desorption ionization (MALDI) is a commonly...
Clinical Trials: Overview
Blinding
You might also read
Related Articles
Articles linked to this work by shared authors, journal, and citation graph.
Leveraging Artificial Intelligence for Clinical Study Matching: Key Threads for Interweaving Data Science and Implementation Science.
Clinical Significance of Marital Status and Changes in Status Extracted from Unstructured Clinical Notes Using Ensembles of Off-the-Shelf Extraction Models.
Large language models for biomedicine: foundations, opportunities, challenges, and best practices.
Natural language processing enabling COVID-19 predictive analytics to support data-driven patient advising and pooled testing.
A Hybrid Model for Family History Information Identification and Relation Extraction: Development and Evaluation of an End-to-End Information Extraction System.
Automating Quality Measures for Heart Failure Using Natural Language Processing: A Descriptive Study in the Department of Veterans Affairs.
Generative Artificial Intelligence Literacy Scale for Nurses: Development and Psychometric Evaluation.
Health Data Quality Skill Gaps and Training Needs Among European Health Data Stakeholders: Cross-Sectional Survey.
Health-Related Digital Engagement and Incident Stroke Among Older Adults: Prospective Cohort Study.
Exploring the Narratives of Patients With Cancer Using Large Language Models: Topic Modeling and Social Network Analysis.
Patients' Experiences of Nurse-Led eHealth Interventions for Chronic Heart Failure: Qualitative Systematic Review and Meta-Synthesis.
Correction: Confidence Measurement Metrics in Multimodal Large Language Models for Ultrasound-Based Radiology Cases: Comparative Evaluation Study of Self-Reported, Consistency-Based, and Hybrid Methods.
Related Experiment Video
Updated: Jun 25, 2025

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
Published on: September 20, 2018
An Extensible Evaluation Framework Applied to Clinical Text Deidentification Natural Language Processing Tools:
Paul M Heider1, Stéphane M Meystre2
1Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, United States.
This study introduces an open-source framework for comparing clinical natural language processing (NLP) deidentification systems. The framework reveals significant performance differences and trade-offs, aiming to advance system evaluation and development.
More Related Videos
Area of Science:
- Computational linguistics
- Medical informatics
- Software engineering
Background:
- Clinical natural language processing (NLP) lacks standardized evaluation frameworks for deidentification systems.
- Current methods offer limited comparability across diverse corpora and personally identifiable information (PII) categories.
- Researchers need a unified system to assess NLP performance consistently.
Purpose of the Study:
- To present an open-source, extensible framework for comparing clinical NLP deidentification system performance.
- To enable cross-corpus evaluations even with non-aligned annotation categories.
- To facilitate easier testing of new systems and corpora within a consistent evaluation pipeline.
Main Methods:
- Developed an end-to-end framework using shell scripts for extensibility.
- Evaluated six off-the-shelf deidentification systems (CliniDeID, deid, MIST, NeuroNER, NLM Scrubber, Philter).
- Tested systems across three clinical text corpora with non-analogous annotation categories, including private and public datasets.
Main Results:
- Observed substantial variations in processing speeds, with MIST being the fastest (24.57 notes/sec) and CliniDeID the slowest (1.00 notes/sec).
- No single system consistently outperformed others; performance varied across PII categories and corpora.
- Identified trade-offs between precision and recall, with CliniDeID and Philter favoring recall, and others favoring precision.
Conclusions:
- Evaluating NLP and deidentification systems in isolation limits comparative analysis.
- A unified evaluation pipeline across multiple systems and corpora enables more nuanced comparisons.
- This open pipeline aims to lower barriers for NLP system evaluation and advancement.

