Transferring Annotator- and Instance-Dependent Transition Matrix for Learning From Crowds.

Area of Science:

• Machine Learning
• Data Science
• Computer Vision

Background:

• Crowd-sourcing data annotation involves multiple annotators, leading to frequent labeling mistakes.
• Modeling label noise using noise transition matrices is crucial for data quality.
• Existing methods struggle with the complexity of annotator- and instance-dependent transition matrices (AIDTM) due to annotation sparsity.

Purpose of the Study:

• To develop a realistic method for estimating general AIDTM.
• To overcome the limitations of existing approaches that assume instance-independence or use simplified parametric models.
• To improve the accuracy and robustness of crowd-sourced data annotation.

Main Methods:

• Parameterizing AIDTM using deep neural networks to maintain modeling generality.
• Employing knowledge transfer by modeling the mixture of annotator noise patterns and transferring this to individual annotators.
• Utilizing knowledge transfer between neighboring annotators to calibrate modeling and mitigate interference between annotators with diverse noise patterns.

Main Results:

• Theoretical analyses confirm the effectiveness of global-to-individual and neighboring-to-neighboring knowledge transfer in modeling AIDTM.
• Experimental results demonstrate the superiority of the proposed approach on both synthetic and real-world crowd-sourcing datasets.
• The method successfully addresses the challenge of annotation sparsity in estimating complex noise transition matrices.

Conclusions:

• The proposed deep neural network-based approach with knowledge transfer effectively models general AIDTM.
• This work provides a more robust solution for handling label noise in crowd-sourced data annotation.
• The findings have significant implications for improving the quality and reliability of machine learning models trained on crowd-sourced data.