Active Multilabel Crowd Consensus.

Area of Science:

• Data Science
• Machine Learning
• Human-Computer Interaction

Background:

• Crowdsourcing offers an economical method for data annotation via online platforms.
• Collecting reliable multilabel data annotations within a limited budget presents significant challenges.
• Existing methods often struggle with worker variability and cost-effectiveness in consensus building.

Purpose of the Study:

• To propose a novel approach, Active Multilabel Crowd Consensus (AMCC), for achieving reliable multilabel data consensus under budget constraints.
• To develop a method that accounts for both shared and individual worker behaviors by grouping workers.
• To introduce an active learning strategy for cost-effective triplet selection (sample-label-worker).

Main Methods:

• AMCC models worker annotations using a linear combination of commonality and individuality, grouping workers with similar behaviors.
• Unreliable workers' impact is minimized by assigning lower weights to their respective groups.
• An active crowdsourcing learning strategy selects informative sample-label-worker triplets for efficient annotation.

Main Results:

• Experimental results show AMCC effectively computes crowd consensus on multilabel datasets.
• AMCC demonstrates superior performance compared to state-of-the-art solutions in achieving consensus.
• The active learning strategy significantly reduces the overall budget by selecting cost-effective triplets.

Conclusions:

• AMCC provides an effective and budget-conscious solution for multilabel data annotation crowdsourcing.
• The proposed method enhances the reliability of crowd consensus by modeling worker behavior.
• AMCC's active learning approach optimizes cost-efficiency in data annotation tasks.