Community annotation experiment for ground truth generation for the i2b2 medication challenge
Ozlem Uzuner1, Imre Solti, Fei Xia
1Department of Information Studies, University at Albany, State University of New York, Albany, NY, USA. ouzuner@albany.edu
Journal of the American Medical Informatics Association : JAMIA
|September 8, 2010
Summary
Community annotators achieved expert-level agreement in clinical record annotation for the i2b2 (Informatics for Integrating Biology and the Bedside) NLP challenge. This demonstrates the value of community efforts for generating high-quality, domain-specific data.
Area of Science:
- Natural Language Processing (NLP)
- Clinical Informatics
- Biomedical Data Science
Background:
- The Third i2b2 Workshop on Natural Language Processing Challenges for Clinical Records aimed to advance NLP in healthcare.
- Community annotation experiments are valuable for generating large-scale, high-quality datasets.
Purpose of the Study:
- To organize and evaluate a community annotation experiment for clinical discharge summaries.
- To assess the reliability and quality of community-generated annotations compared to expert annotations.
Main Methods:
- Released annotation guidelines and a small set of annotated discharge summaries.
- Participants annotated 10 summaries each; each summary was annotated by three annotators, with disagreements resolved by a third.
- Inter-annotator agreement was measured for community and expert annotators, with experts annotating both raw records and pooled system outputs.
Main Results:
- Community annotators achieved inter-annotator agreement comparable to expert annotators.
- Community-generated ground truth achieved F-measures above 0.90 when compared to expert ground truth.
- High-quality ground truth was generated by the community even for complex, domain-specific tasks.
Conclusions:
- Community annotation is a reliable and cost-effective method for creating high-quality clinical NLP datasets.
- The i2b2 team's experiment validated the use of crowdsourced data for intricate annotation tasks.
- Findings support the broader adoption of community annotation in clinical informatics research.
