Ground Truth Creation for Complex Clinical NLP Tasks - an Iterative Vetting Approach and Lessons Learned

Area of Science:

• Clinical Informatics
• Natural Language Processing (NLP)
• Medical Data Analysis

Background:

• Natural Language Processing (NLP) offers potential for analyzing electronic health records (EHRs) to reduce physician cognitive load.
• High-quality "ground truth" datasets are essential for training and validating clinical NLP algorithms.
• Creating ground truth for complex medical data is challenging due to the vast amount of information physicians must process.

Purpose of the Study:

• To present an iterative vetting methodology for creating ground truth datasets for complex NLP tasks in healthcare.
• To evaluate the impact of this methodology on ground truth quality and system accuracy for automated problem list generation.

Main Methods:

• Developed and implemented an iterative vetting approach for ground truth data creation.
• Applied the methodology to an automated problem list generation task using EHR data.
• Assessed the quality of the generated ground truth and the accuracy of the NLP system.

Main Results:

• The iterative vetting approach was successfully applied to create ground truth for a complex NLP task.
• The methodology demonstrated a positive effect on the quality of the ground truth data.
• Improved ground truth quality led to enhanced accuracy in the automated problem list generation system.

Conclusions:

• An iterative vetting process is an effective strategy for mitigating inaccuracies in ground truth creation for clinical NLP.
• This approach enhances the reliability of datasets used for training and testing medical NLP algorithms.
• Lessons learned from this effort provide valuable insights for future clinical NLP development.