CrowdAttention: An Attention Based Framework to Classify Crowdsourced Data in Medical Scenarios

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Supervised learning relies on high-quality labeled data, which is expensive and time-consuming to obtain.
• Crowdsourcing offers a scalable solution but introduces label noise due to non-expert annotator variability.
• Existing multi-annotator learning methods struggle to effectively address this inherent noise.

Purpose of the Study:

• To propose CrowdAttention, a novel deep learning framework for robust classification using noisy crowdsourced labels.
• To jointly model data classification and annotator reliability within a single end-to-end architecture.
• To enhance the accuracy and robustness of models trained on heterogeneous, non-expert annotations.

Main Methods:

• Developed an end-to-end deep learning framework, CrowdAttention, integrating classification and annotator reliability modeling.
• Utilized a cross-attention mechanism to couple two networks: one for classification and one for annotator reliability scoring.
• The crowd network assigns instance-dependent reliability scores based on annotator label alignment with model predictions.

Main Results:

• Demonstrated improved classification accuracy on both synthetic and real-world datasets.
• Showcased enhanced robustness against label noise compared to existing state-of-the-art methods.
• Validated the effectiveness of jointly modeling classification and annotator reliability.

Conclusions:

• CrowdAttention provides an effective solution for leveraging crowdsourced data despite label noise.
• The proposed cross-attention mechanism successfully models annotator reliability, leading to superior performance.
• This framework offers a significant advancement for supervised learning in domains reliant on non-expert annotations.