Towards Globally Optimal Crowdsourcing Quality Management: The Uniform Worker Setting

Area of Science:

• Artificial Intelligence
• Data Science
• Human-Computer Interaction

Background:

• Crowdsourcing quality management aims to determine true task answers and worker reliability from responses.
• Existing methods, like Expectation-Maximization (EM), often yield only locally optimal solutions.
• A need exists for algorithms guaranteeing globally optimal results in crowdsourcing quality assessment.

Purpose of the Study:

• To develop algorithms for globally optimal crowdsourcing quality management.
• To address filtering (yes/no) and rating (integer scores) task types.
• To analyze the computational complexity of these novel algorithms.

Main Methods:

• Designed algorithms to find global optimal estimates for task answers and worker quality.
• Conceptually explored all possible task-to-answer mappings.
• Utilized two key strategies to drastically reduce the search space while maintaining optimality.

Main Results:

• Developed algorithms that guarantee globally optimal solutions for crowdsourcing quality management.
• Significantly reduced the number of mappings considered, improving computational efficiency.
• Demonstrated superior accuracy compared to existing EM-based algorithms.

Conclusions:

• This work provides a significant advancement in understanding the complexity of achieving globally optimal crowdsourcing quality management.
• The proposed algorithms offer a more accurate and efficient approach to assessing crowdsourced data quality.
• Highlights the potential for globally optimal solutions in complex crowdsourcing scenarios.