Challenges in clinical natural language processing for automated disorder normalization.

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Biomedical Research

Background:

• Identifying disorders in electronic health records (EHRs) is crucial for clinical practice and research.
• Named Entity Recognition (NER) and normalization of disorders perform less effectively in clinical narratives compared to biomedical publications.
• This study investigates the reasons for this performance gap and proposes solutions.

Purpose of the Study:

• To identify the causes of reduced performance in disorder NER and normalization within clinical narratives.
• To introduce generalizable solutions to improve the accuracy of disorder identification in EHRs.
• To develop and evaluate a system (DNorm-C) for disorder NER and normalization in clinical text.

Main Methods:

• Compared vocabulary richness in clinical narratives versus biomedical publications using closure properties.
• Employed machine learning for disorder NER using linear-chain conditional random fields with enhanced lexical features.
• Utilized pairwise learning to rank for normalization, automatically learning term variations from data.

Main Results:

• Clinical narratives exhibit richer disorder terminology and higher term variation than biomedical publications.
• The DNorm-C system achieved an F-score of 0.753 for NER and 0.672 for normalization on ShARe/CLEF eHealth Task data.
• Improvements increased NER F-score by 0.039 and normalization F-score by 0.036; a high-recall NER version boosted normalization recall to 0.744.

Conclusions:

• High term variation in clinical narratives is a primary reason for reduced NER and normalization performance.
• Pairwise learning to rank and lexical enhancements effectively address term variation in clinical NER.
• DNorm-C is an open-source, high-performing system for disorder identification in clinical text, advancing NER and normalization capabilities.