Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing

Area of Science:

• Biomedical Natural Language Processing
• Machine Learning in Healthcare
• Data Annotation Strategies

Background:

• High-quality gold standards are essential for supervised machine learning in clinical natural language processing (NLP).
• Traditional expert annotation is costly and time-consuming.
• Crowdsourcing offers a potentially cheaper alternative but has shown variable quality in biomedical domains.

Purpose of the Study:

• To investigate the feasibility and quality of crowdsourcing for clinical NLP tasks.
• To achieve high agreement between crowdsourced and traditionally developed gold standards.
• To evaluate crowdsourcing for annotating medication names, types, and attributes in clinical trial announcements.

Main Methods:

• 1042 clinical trial announcements (CTAs) were randomly selected and double-annotated.
• Crowdsourcing was performed using the CrowdFlower platform.
• Sensitivity, precision, and F-measure were calculated to evaluate annotation quality; chi-square tests assessed statistical significance.

Main Results:

• High agreement (F-measure 0.87 for names, 0.73 for types) was achieved between crowdsourced and traditional annotations.
• Excellent agreement (0.96 F-measure) was observed for linking medications with attributes.
• Simple voting emerged as the optimal aggregation method, with no statistically significant difference from traditional corpora.

Conclusions:

• Crowdsourcing is a feasible, inexpensive, and practical method for collecting high-quality clinical text annotations, excluding protected health information.
• Well-designed interfaces and rigorous quality control are critical for successful crowdsourcing.
• The study will release annotation infrastructure code and generated corpora to facilitate future research.