Extracting and classifying diagnosis dates from clinical notes: A case study.

Area of Science:

• Hematology
• Medical Informatics
• Natural Language Processing

Background:

• Myeloproliferative neoplasms (MPNs) are chronic blood cancers requiring long-term management.
• Accurate original diagnosis dates are crucial for patient care and research but are often inconsistently documented in electronic health records.
• Challenges exist in extracting and classifying relevant dates from unstructured clinical text.

Purpose of the Study:

• To develop and evaluate a natural language processing (NLP) pipeline for extracting and classifying diagnosis dates from clinical notes.
• To assess the performance of NLP tools, specifically lightweight Python packages and a rules-based regular expression (regex) approach, in identifying and interpreting clinical dates.

Main Methods:

• An NLP pipeline was designed to extract dates from clinical text.
• Lightweight Python packages were evaluated for date identification and interpretation.
• A rules-based regular expression (regex) approach was implemented and tested.
• Performance was measured using recall metrics on manually annotated diagnosis dates.

Main Results:

• Existing lightweight NLP packages demonstrated low specificity in handling partial and relative dates.
• The rules-based regex approach achieved 83.0% recall for diagnosis dates and 77.4% for all annotated dates.
• Initial MPN diagnoses represented only 3.8% of annotated dates, indicating significant class imbalance and noise.

Conclusions:

• Automated extraction of clinical diagnosis dates using NLP presents significant challenges, particularly with partial and relative dates.
• A regex-based method offers a viable, albeit imperfect, solution for date extraction in this context.
• Addressing class imbalance and improving targeting of relevant date instances are critical for enhancing the accuracy of future NLP pipelines for MPN research.