Assessing the Performance of Clinical Natural Language Processing Systems: Development of an Evaluation Methodology

Area of Science:

• Computer Science
• Medical Informatics
• Natural Language Processing

Background:

• Clinical natural language processing (cNLP) systems extract valuable information from electronic health records (EHRs).
• Structured data from EHRs enhances clinical knowledge generation and decision-making.
• Robust evaluation of cNLP systems is challenging due to a lack of standardized guidance.

Purpose of the Study:

• To provide NLP experts with a structured methodology for evaluating cNLP systems.
• To ensure the robustness and representativeness of cNLP system performance metrics.
• To offer a systematic approach for the complex task of cNLP system evaluation.

Main Methods:

• A five-phase evaluation methodology: target population definition, statistical document collection, annotation guideline design, external annotation, and performance evaluation.
• Introduction of the Sample Size Calculator for Evaluations (SLiCE) software tool for determining optimal gold standard size.
• Application of the methodology to evaluate the EHRead Technology cNLP system on asthma patient data.

Main Results:

• The proposed methodology effectively guided cNLP system evaluation in a real-world asthma patient study.
• SLiCE facilitated the creation of a meaningful gold standard by calculating the necessary document count.
• Performance metrics were obtained within expected confidence intervals using only 519 EHRs.

Conclusions:

• The five-phase methodology provides essential guidance for NLP experts evaluating cNLP systems.
• This approach enhances evaluation robustness and prevents inefficient use of resources.
• SLiCE is offered as an accessible, open-source Python library to support these evaluations.