Robust Cumulative Crowdsourcing Framework Using New Incentive Payment Function and Joint Aggregation Model.

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• Crowdsourcing is vital for labeled data in machine learning.
• Crowdsourced labels are often unreliable due to inflexible mechanisms, poor incentives, and inexpert workers.

Purpose of the Study:

• To propose a robust crowdsourcing framework addressing unreliable labels.
• To enhance data quality and worker engagement in crowdsourcing tasks.

Main Methods:

• Introduced a flexible data collection mechanism using cumulative voting.
• Designed a theoretically incentive-compatible payment function.
• Developed efficient aggregation models, including simplex constrained majority voting (SCMV), enhanced with probabilistic generative models.

Main Results:

• Collected crowd labels with higher quality without increased cost.
• Aggregation models outperformed state-of-the-art methods in accuracy and convergence speed.
• Demonstrated effectiveness across multiple crowdsourcing datasets.

Conclusions:

• The proposed framework offers a comprehensive solution for noisy crowdsourced labels.
• The novel components improve data collection, worker incentives, and label aggregation.
• This approach leads to more accurate and efficient machine learning model training.