Related Experiment Videos
Producing knowledge by admitting ignorance: Enhancing data quality through an "I don't know" option in citizen
Marina Torre1, Shinnosuke Nakayama1, Tyrone J Tolbert1
1Department of Mechanical and Aerospace Engineering, New York University Tandon School of Engineering, Brooklyn, New York, United States of America.
Plos One
|February 28, 2019
Summary
Adding an "I don't know" option to citizen science projects improves data accuracy. This helps identify environmental threats more reliably by providing meaningful information from non-expert volunteers.
Area of Science:
- Environmental Science
- Data Science
- Human-Computer Interaction
Background:
- The "noisy labeler problem" is a significant challenge in crowdsourced data, impacting citizen science projects.
- Dynamic task allocation, using entropy, aims to improve data quality by updating labeler agreement.
- Citizen science relies on non-experts to generate high-quality data for scientific research.
Purpose of the Study:
- To investigate if providing an "I don't know" option enhances data quality in citizen science.
- To assess the impact of "I don't know" votes on the agreement and accuracy of labeler classifications.
- To explore the information-theoretic value of "blank votes" in crowdsourced data.
Main Methods:
- A citizen science project classifying canal images as "threat" or "no threat" was used.
- The study compared data quality with and without an "I don't know" option for volunteers.
- Information-theoretic concepts, specifically entropy, were used to analyze labeler agreement.
Main Results:
- The inclusion of an "I don't know" option significantly enhanced overall data accuracy.
- The improvement in accuracy was primarily observed in the true negative rate, not the true positive rate.
- "Blank votes" were found to contain meaningful information for improving data accuracy.
Conclusions:
- Providing an "I don't know" option is an effective strategy for improving data quality in citizen science.
- Even seemingly non-informative votes can provide valuable insights in crowdsourced data analysis.
- This finding has important implications for the design of citizen science platforms and data collection methods.