Online label aggregation with incomplete crowd responses.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Traditional crowdsourcing aggregation methods struggle with asynchronous and incomplete response streams.
• Existing online methods often require complete data per step or inefficiently reload historical responses, posing storage and privacy challenges.

Purpose of the Study:

• To develop an online label aggregation framework, OLA-Incomplete, specifically designed for incomplete response streams in crowdsourcing.
• To address limitations of prior methods by mitigating catastrophic forgetting and modeling unknown worker reliability.

Main Methods:

• Introduced OLA-Incomplete, integrating a variational-inference aggregator with a generative replay module.
• The generative replay module preserves historical information without reloading, mitigating forgetting by replaying past data.
• The aggregator infers truths by maximizing evidence lower bound over replayed and new labels, explicitly modeling worker reliability.

Main Results:

• Achieved high final accuracies on public datasets: 90.74% (Duck), 92.50% (RTE), and 95.99% (PostSent).
• Demonstrated at least 7.79% relative improvement over the strongest baseline.
• Exhibited strong instantaneous online accuracy and robustness to varying response chunk sizes and arrival orders.

Conclusions:

• OLA-Incomplete offers a practical and effective solution for real-world crowdsourcing workflows with incomplete data.
• The framework's ability to handle asynchronous, incomplete responses and model worker reliability enhances data aggregation accuracy and efficiency.