Evaluation of a Concept Mapping Task Using Named Entity Recognition and Normalization in Unstructured Clinical Text.

Area of Science:

• Clinical Informatics
• Natural Language Processing

Background:

• Extracting structured clinical concepts from unstructured free text is challenging.
• Automated methods are needed to improve efficiency and accuracy in clinical data mining.

Purpose of the Study:

• To evaluate the feasibility and accuracy of a commercial natural language processing (NLP) engine for clinical concept extraction.
• To assess the performance of the NLP engine in named entity recognition and normalization tasks from clinical letters.

Main Methods:

• A commercial NLP engine (Linguamatics I2E v5.3.1) was used with an Intelligent Medical Objects ontology.
• Sixty anonymized clinical letters were manually annotated by clinicians to create a gold standard.
• Performance was measured using precision, recall, and F1 scores, with subset analysis for various factors.

Main Results:

• The NLP engine achieved F1 scores of 0.81 (strict) and 0.84 (relaxed) when negation was not considered.
• F1 scores were 0.75 (strict) and 0.77 (relaxed) when accurate negation was required.
• Higher F1 scores were observed for concepts extracted from continuous text.

Conclusions:

• A commercially available NLP engine demonstrates good performance in extracting a wide range of clinical concepts from free text.
• This technology shows promise for populating problem lists and supporting clinical data mining projects.