Improving Crowdsourced Label Quality Using Noise Correction.

Area of Science:

• Data Science
• Machine Learning
• Crowdsourcing

Background:

• Crowdsourcing systems offer cost-effective data labeling but struggle with label quality assurance.
• Multiple noisy labels from various sources often require sophisticated integration and correction methods.

Purpose of the Study:

• To propose a novel framework for improving integrated label quality in crowdsourcing by employing noise correction techniques.
• To introduce and evaluate an adaptive voting noise correction (AVNC) algorithm for precise identification and correction of noisy labels.

Main Methods:

• A general framework utilizing front-end ground truth inference to estimate labeler quality.
• Supervision of label noise filtering and correction based on estimated labeler performance.
• Development of the adaptive voting noise correction (AVNC) algorithm to identify and correct noisy labels.
• Creation of weak classifiers from cleansed data, leading to a powerful ensemble classifier for noise correction.

Main Results:

• The proposed framework consistently improves label quality across diverse simulated and real-world datasets, especially when instances have repeated labels.
• The AVNC algorithm demonstrates superior performance compared to state-of-the-art methods by considering both the quantity and probability of label noise.
• The framework's effectiveness is independent of the specific inference algorithms used.

Conclusions:

• The developed framework significantly enhances the quality of integrated labels derived from noisy crowdsourced data.
• The adaptive voting noise correction (AVNC) algorithm offers a robust and effective solution for addressing label noise in crowdsourcing.
• This approach provides a valuable tool for improving the reliability of data used in machine learning models.