Performance of Natural Language Processing versus International Classification of Diseases Codes in Building...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Public Health Surveillance

Background:

• Standardized health registries often rely on administrative codes (e.g., International Classification of Diseases - ICD) for patient data.
• Current coding practices may misclassify fall injuries, necessitating manual data review for accuracy.
• This highlights a need for more precise methods to capture fall-related incidents in patient records.

Purpose of the Study:

• To develop and evaluate natural language processing (NLP) models for automated extraction of fall incidents and mechanisms from clinical notes.
• To compare the performance of NLP methods against traditional ICD coding for fall data.
• To improve the accuracy and efficiency of data collection for fall injury registries.

Main Methods:

• Retrospective review of clinical notes from patients with hip fractures, categorized into fall-induced (case) and non-fall-induced (control) groups.
• Development of NLP models for two tasks: fall occurrence detection and fall mechanism classification.
• Performance evaluation using metrics such as accuracy, sensitivity, specificity, and F1-score, compared against ICD codes.

Main Results:

• NLP models achieved high accuracy in detecting fall occurrences (F1-score=0.97) and classifying fall mechanisms (F1-score=0.61).
• NLP demonstrated superior performance over ICD codes, detecting 98% of fall occurrences versus 26% and 65% of fall mechanisms versus 12%.
• The study analyzed 1769 notes for fall occurrence and 783 for mechanism classification.

Conclusions:

• NLP algorithms offer a more accurate and efficient method for identifying fall occurrences and mechanisms in clinical notes compared to ICD codes.
• This automated approach can significantly enhance the quality of data in disease registries, particularly for fall-related injuries.
• The proposed NLP methodology holds potential for application in other large-scale data registries requiring accurate annotation and classification.