A taxonomy for advancing systematic error analysis in multi-site electronic health record-based clinical concept...

Area of Science:

• Clinical Natural Language Processing (NLP)
• Health Informatics
• Medical Data Analysis

Background:

• Error analysis is vital for refining clinical concept extraction in NLP.
• Current methods face challenges in standardization due to EHR heterogeneity.
• Manual error review requires NLP and domain expertise, complicating reproducibility.

Purpose of the Study:

• To establish common definitions and taxonomies for clinical concept extraction errors.
• To foster community consensus on error analysis procedures.
• To improve the standardization and reproducibility of NLP error analysis.

Main Methods:

• Iterative development and evaluation of an error taxonomy.
• Incorporation of existing literature, standards, real-world data, and multisite evaluations.
• Community feedback and release of the taxonomy in .dtd and .owl formats compatible with annotation tools.

Main Results:

• A taxonomy with 43 error classes across 6 dimensions and 4 properties was created.
• Evaluations showed significant variations in error types based on methodology, tasks, and EHR settings.
• Community feedback highlighted the need for enhanced clarity, generalizability, usability, and dissemination.

Conclusions:

• The proposed taxonomy standardizes and accelerates multi-site error analysis for clinical NLP.
• It improves the provenance, interpretability, and portability of NLP models.
• Future work may focus on developing automated methods for error classification and standardization.