Accurate inference of crowdsourcing properties when using efficient allocation strategies.

Area of Science:

• Crowdsourcing
• Machine Learning
• Data Science

Background:

• Allocation strategies enhance crowdsourcing efficiency by optimizing task assignment.
• These strategies can introduce bias, leading to non-representative datasets.
• This bias complicates the accurate inference of crucial crowd and task properties.

Purpose of the Study:

• To develop a method for inferring crowd and task properties from biased crowdsourcing data.
• To address the challenge of non-representative datasets caused by allocation algorithms.
• To enable accurate inference while retaining efficiency gains from allocation strategies.

Main Methods:

• Introduction of Decision-Explicit Probability Sampling (DEPS).
• DEPS accounts for potential bias introduced by allocation strategies.
• Experimental validation on both real and synthetic crowdsourcing datasets.

Main Results:

• DEPS outperforms existing baseline inference methods.
• The method successfully leverages efficiency gains from allocation strategies.
• Accurate inference of general properties is achieved even with non-representative data.

Conclusions:

• DEPS provides a robust solution for biased crowdsourcing data.
• It allows for more comprehensive knowledge extraction from crowdsourced datasets.
• This facilitates better understanding of task difficulty and worker performance.