Heuristic sample selection to minimize reference standard training set for a part-of-speech tagger

Area of Science:

• Natural Language Processing (NLP)
• Computational Linguistics
• Medical Informatics

Background:

• Part-of-speech (POS) tagging is crucial for medical NLP.
• General English corpora yield poor performance on specialized medical text.
• Creating annotated medical corpora is labor-intensive and time-consuming.

Purpose of the Study:

• To investigate a heuristic-based sample selection method to minimize annotated corpus size for retraining a Maximum Entropy (ME) POS tagger.
• To reduce human annotation requirements for training medical NLP systems.

Main Methods:

• Developed a manually annotated domain-specific corpus (DSC) of surgical pathology reports and a domain-specific lexicon (DL).
• Sampled the DSC using two heuristics to create smaller training sets.
• Compared retrained ME POS tagger performance against general-trained ME tagger, DL-retrained ME tagger, and MedPost tagger.

Main Results:

• The ME tagger retrained with the DSC outperformed taggers retrained with the DL and the MedPost tagger.
• Heuristic sample selection achieved performance equivalent to using the entire training set but with significantly fewer sentences.
• Learning curve analysis indicated an 84% reduction in training set size was possible without performance loss.

Conclusions:

• Heuristic sample selection effectively minimizes human annotation efforts for training medical NLP systems.
• This method enables the development of accurate POS taggers for medical text with reduced resource investment.
• Domain-specific corpora combined with heuristic sampling offer a scalable solution for medical NLP model training.