Related Experiment Videos
Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction
Aurélie Névéol1, Rezarta Islamaj Doğan, Zhiyong Lu
1National Center for Biotechnology Information, US National Library of Medicine, 8600 Rockville Pike, Bethesda, MD 20894, USA.
Journal of Biomedical Informatics
|November 25, 2010
Summary
Automated pre-annotation significantly reduces manual effort for biomedical query annotation, cutting hand annotations by 28.9%. This speeds up the process and improves consistency for large-scale data projects.
Area of Science:
- Biomedical Informatics
- Natural Language Processing
- Data Annotation
Background:
- Information processing algorithms need extensive annotated data, but its creation is costly and complex.
- Semantic annotation of biomedical queries is crucial for training and testing algorithms.
Purpose of the Study:
- To evaluate the benefits of a state-of-the-art tool for semantic annotation of biomedical queries.
- To assess the impact of automatic pre-annotation on annotation efficiency, quality, and consistency.
Main Methods:
- Seven annotators semantically annotated 10,000 PubMed queries across 16 categories.
- Queries were either annotated from scratch or manually corrected after automatic pre-annotation.
- Impact was measured by time, actions, satisfaction, agreement, and annotation quality.
Main Results:
- Automatic pre-annotations reduced required hand annotations by 28.9%.
- Annotation time decreased substantially, and inter-annotator agreement significantly increased.
- No significant difference was observed in semantic distribution or the number of annotations produced.
Conclusions:
- Automatic pre-annotation tools are beneficial for large-scale manual annotation projects.
- These tools accelerate annotation time and enhance consistency while maintaining high-quality results.
- The study recommends using automated tools to assist in biomedical data annotation efforts.
