Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Crowd control: Effectively utilizing unscreened crowd workers for biomedical data annotation.

Anne Cocos1, Ting Qian2, Chris Callison-Burch3

  • 1Department of Biomedical and Health Informatics, The Children's Hospital of Philadelphia, United States; Computer and Information Sciences Department, University of Pennsylvania, United States.

Journal of Biomedical Informatics
|April 9, 2017
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

HMGB3 facilitates malignant phenotype of esophageal carcinoma via histone lactylation- and Wnt/β-catenin-dependent transcription activation of c-myc.

NPJ precision oncology·2026
Same author

Human-AI Cooperation in Healthcare and Rehabilitation.

Delaware journal of public health·2026
Same author

Efficacy and safety analysis of immune checkpoint inhibitor rechallenge in advanced esophageal squamous cell carcinoma: A retrospective study.

Human vaccines & immunotherapeutics·2025
Same author

A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis.

Advances in neural information processing systems·2025
Same author

Predicting explainable dementia types with LLM-aided feature engineering.

Bioinformatics (Oxford, England)·2025
Same author

Automated Shared Phenotype Discovery in Undiagnosed Cohorts for Rare Disease Research.

Proceedings of the ... International Conference on Machine Learning and Applications. International Conference on Machine Learning and Applications·2025

Crowdsourcing annotations from online workers is as effective as expert annotation for training machine learning models on Electronic Health Records (EHR) data. Self-reported confidence levels help identify low-quality annotations.

Area of Science:

  • Biomedical Informatics
  • Natural Language Processing
  • Machine Learning

Background:

  • Manual annotation of Electronic Health Records (EHR) data is crucial for machine learning research but is becoming impractical due to data volume.
  • Expert annotation ensures high-quality data but faces scalability challenges with increasing EHR data.

Purpose of the Study:

  • To evaluate the effectiveness of crowdsourcing with unscreened online workers for annotating unstructured EHR text.
  • To determine if crowdsourced annotations can be used effectively in supervised learning models.
  • To assess the utility of self-reported worker confidence levels in improving annotation quality.

Main Methods:

  • Crowdsourcing unstructured EHR text from unscreened online workers.
  • Training a sentence classification model using both expert-annotated and crowdsourced-annotated data.
Keywords:
CrowdsourcingEHR dataLogistic regressionSentence classificationText annotations

Related Experiment Videos

  • Analyzing the performance of the model trained on each dataset.
  • Investigating the impact of worker-reported confidence levels on annotation accuracy.

    • Main Results:

    • Crowdsourced annotation data demonstrated comparable effectiveness to expert data in training a sentence classification model.
    • The model successfully detected mentions of abnormal ear anatomy in audiology radiology reports.
    • Self-reported confidence levels enabled the identification of less accurate annotations for expert review.

    Conclusions:

    • Unscreened online crowd workers can effectively contribute to annotating unstructured EHR datasets.
    • Crowdsourcing offers a scalable alternative to manual expert annotation for EHR data.
    • Incorporating confidence scores enhances the reliability of crowdsourced annotations.