Active Crowdsourcing for Multilabel Annotation.

Area of Science:

• Machine Learning
• Data Science
• Computer Vision

Background:

• Multilabel annotation is crucial for training classification models but is often time-consuming and expensive when relying on domain experts.
• Crowdsourcing offers a faster alternative for acquiring labels but presents challenges related to data acquisition costs and label quality.

Purpose of the Study:

• To propose a novel active crowdsourcing method, One-Coin Label-Dependent Active Crowdsourcing (OCLDAC), for efficient and accurate multilabel annotation.
• To enhance the truth inference of noisy labels by modeling label correlations.
• To optimize the selection of instances, labels, and workers for improved model training.

Main Methods:

• Developed a novel one-coin label-dependent model using a mixture of independent Bernoulli distributions to infer integrated labels.
• Implemented active learning strategies that consider noisy label distribution, model prediction probability, label correlations, and worker reliability.
• Conducted simulations on eight multilabel datasets and evaluated on a real-world crowdsourcing dataset.

Main Results:

• The proposed OCLDAC method demonstrated significant performance improvements over state-of-the-art methods.
• The integrated label inference model effectively leveraged label correlations to enhance truth inference accuracy.
• The novel selection strategies optimized the crowdsourcing process for better data acquisition.

Conclusions:

• OCLDAC offers a more accurate and efficient solution for multilabel annotation compared to existing methods.
• The method effectively addresses the challenges of cost and quality in crowdsourced data acquisition for machine learning.
• OCLDAC is a promising approach for generating high-quality training sets in the era of big data and rapidly evolving models.