Pooling annotated corpora for clinical concept extraction.

Area of Science:

• Natural Language Processing (NLP)
• Medical Informatics
• Machine Learning

Background:

• Annotated corpora are crucial for machine learning in clinical concept extraction.
• Creating these annotations is costly and labor-intensive.
• Reusing existing corpora across institutions through pooling offers a potential cost-effective alternative.

Purpose of the Study:

• To evaluate the effectiveness of pooling corpora from different institutions for training machine taggers.
• To investigate the impact of differing annotation guidelines on tagger performance.
• To identify factors influencing the success of corpus pooling for medical problem recognition.

Main Methods:

• Utilized corpora from the 2010 i2b2/VA NLP challenge and Mayo Clinic Rochester.
• Employed MedTagger, a system incorporating dictionary lookup, part-of-speech tagging, and machine learning.
• Examined annotation guidelines to understand differences and their effects on pooling.

Main Results:

• Corpus pooling proved effective when the local corpus was small and annotation guidelines were reconciled.
• The benefits of pooling diminished as the size of the local training data increased.
• Annotation guideline compatibility, report type distribution, and corpus sizes were identified as key factors.

Conclusions:

• The success of pooling annotated corpora depends on guideline compatibility, report diversity, and corpus sizes.
• Simple methods to reconcile guideline differences can enhance pooling effectiveness.
• Recommendations include developing standard annotation guidelines, using a two-pass annotation method, and creating metadata for corpora.