Aggregating soft labels from crowd annotations improves uncertainty estimation under distribution shift.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Expert annotations for machine learning are costly, while crowd-sourced labels can be unreliable.
• Learning from crowd label distributions (soft labels) shows promise for performance and uncertainty estimation.
• Existing studies primarily focus on in-domain settings with limited soft-labeling methods.

Purpose of the Study:

• To conduct a large-scale empirical study of soft-labeling methods for crowd-sourced data in out-of-domain settings.
• To evaluate 8 different soft-labeling methods across 4 language and vision tasks.
• To propose and validate a simple averaging aggregation method for soft labels.

Main Methods:

• Systematic analysis of 8 soft-labeling methods on 4 diverse language and vision tasks.
• Implementation of a simple averaging strategy to aggregate soft labels.
• Comparison of the aggregation method against individual soft-labeling methods and majority voting.

Main Results:

• Averaging soft labels consistently improves predictive uncertainty estimation across most settings.
• The proposed aggregation method maintains competitive raw performance compared to other approaches.
• Method selection is less critical with abundant or minimal data, but aggregation significantly boosts uncertainty for subjective labels with moderate data.

Conclusions:

• Simple averaging of soft labels provides a robust and consistent approach for learning from crowd-sourced annotations.
• This aggregation strategy enhances model uncertainty estimation, particularly valuable for subjective tasks.
• The findings offer practical guidance for selecting and applying crowd-labeling techniques in machine learning.