Scaling drug indication curation through crowdsourcing.

Area of Science:

• Bioinformatics
• Computational Biology
• Pharmacology

Background:

• Manual curation of biological databases is costly and time-consuming.
• Existing methods for cataloging drug indications from FDA drug labels require substantial human effort (e.g., LabeledIn took 40+ hours).
• There is a need for scalable and cost-effective approaches to biological data annotation.

Purpose of the Study:

• To investigate the feasibility of using crowdsourcing for drug indication annotation.
• To assess the scalability, cost-effectiveness, and accuracy of crowdsourcing via Amazon Mechanical Turk (MTurk).
• To develop a simplified task interface for non-expert workers to annotate drug indications.

Main Methods:

• Transformed expert-curation task into binary judgment Human Intelligence Tasks (HITs) on MTurk.
• Designed a novel crowdsourcing interface encoding annotation guidelines into user options.
• Deployed over 3000 HITs across 706 drug labels and collected 18,775 judgments from 74 workers.

Main Results:

• Collected data within 8 hours of posting HITs.
• Achieved an aggregated accuracy of 96% on control HITs.
• Obtained results at a cost of $1.75 per drug label, demonstrating significant cost and time savings.

Conclusions:

• Crowdsourcing drug indication annotation is a feasible and efficient alternative to expert curation.
• The proposed method achieves accuracy comparable to domain experts.
• This approach offers substantial cost and time savings for biological database curation.