Microtask crowdsourcing for disease mention annotation in PubMed abstracts

Area of Science:

• Biomedical Natural Language Processing (BioNLP)
• Computational Biology
• Data Annotation

Background:

• Developing large, annotated corpora is crucial for advancing BioNLP and machine learning in biomedical text analysis.
• Traditional expert annotation is time-consuming and costly.
• Microtask crowdsourcing platforms show potential for generating high-quality annotations.

Purpose of the Study:

• To investigate the efficacy of Amazon Mechanical Turk (AMT) for annotating disease mentions in PubMed abstracts.
• To refine a crowdsourcing protocol for accurate biomedical text annotation.
• To benchmark the protocol against the NCBI Disease corpus.

Main Methods:

• Utilized AMT workers to annotate disease mentions in 593 PubMed abstracts.
• Developed and iterated on a crowdsourcing protocol.
• Merged annotations using a simple voting method.
• Benchmarked results against the NCBI Disease corpus gold standard.

Main Results:

• Achieved an overall F-measure of 0.872 (precision 0.862, recall 0.883) compared to the gold standard.
• Annotation quality, measured by F-measure, improved with more workers per task, with diminishing returns beyond 8 workers.
• Successfully annotated 593 documents in 9 days with 145 workers at a low cost per abstract.

Conclusions:

• Microtask crowdsourcing via AMT is a viable and efficient method for generating well-annotated biomedical corpora.
• This approach can significantly accelerate the creation of datasets for BioNLP research.
• The developed protocol demonstrates the potential of crowdsourcing for large-scale biomedical text annotation.