An Interactive Method to Improve Crowdsourced Annotations.

Area of Science:

• Machine Learning
• Data Science
• Human-Computer Interaction

Background:

• Crowdsourced annotations are prone to noise, necessitating validation.
• Existing learning-from-crowds models lack efficient expert validation facilitation.
• Expert validation is crucial for accurate label inference from noisy data.

Purpose of the Study:

• To develop an interactive method for efficient expert validation of crowdsourced annotations.
• To assist experts in identifying and verifying uncertain instance labels and unreliable workers.
• To integrate visualizations for improved understanding of annotation quality and model performance.

Main Methods:

• An interactive learning-from-crowds framework was developed.
• Candidate instances and workers for validation were selected based on model inference.
• Confusion visualization, constrained projection, and scatter-plot visualization were employed.
• Verified results were propagated through the learning-from-crowds model iteratively.

Main Results:

• The proposed method efficiently assists experts in validating crowdsourced annotations.
• Visualizations provide insights into confusing classes, uncertain labels, and worker reliability.
• The integrated system offers an iterative and progressive environment for data validation.
• Case studies demonstrate the effectiveness in improving annotation quality.

Conclusions:

• The interactive validation method significantly enhances the efficiency of processing noisy crowdsourced data.
• Integrated visualizations aid experts in making informed validation decisions.
• The approach effectively improves the accuracy of labels inferred from crowdsourced datasets.