Feasibility of pooling annotated corpora for clinical concept extraction

Area of Science:

• Clinical Natural Language Processing (NLP)
• Machine Learning in Healthcare
• Medical Informatics

Background:

• Annotated corpora are essential for training machine learning models for clinical concept extraction.
• Creating these corpora is resource-intensive for individual institutions.
• Pooling corpora from multiple sources is a potential strategy to overcome resource limitations.

Purpose of the Study:

• To investigate if pooling corpora from different sources improves the performance and portability of machine learning taggers for medical problem detection.
• To evaluate the impact of combining data from the 2010 i2b2/VA NLP challenge and Mayo Clinic Rochester datasets.

Main Methods:

• Utilized machine learning taggers trained on pooled corpora from two distinct sources: the 2010 i2b2/VA NLP challenge and Mayo Clinic Rochester.
• Evaluated tagger performance using F1-score for medical problem recognition.
• Examined annotation guidelines to identify sources of incompatibility between corpora.

Main Results:

• Contrary to expectations, pooling corpora resulted in a decrease in the F1-score, indicating reduced performance.
• Differences in annotation guidelines were identified as a key factor contributing to corpus incompatibility.
• The study highlights challenges in achieving effective model portability when combining diverse datasets.

Conclusions:

• Pooling annotated clinical corpora from different institutions may not improve, and can even decrease, the performance of machine learning models for medical problem detection.
• Standardized annotation guidelines are necessary for the clinical NLP community to ensure corpus compatibility and enhance model portability.
• Future efforts should focus on developing consensus guidelines to facilitate effective data sharing and improve NLP tool development in healthcare.