Learning by aggregating experts and filtering novices: a solution to crowdsourcing problems in bioinformatics

Area of Science:

• Biomedical applications
• Machine learning
• Data science

Background:

• Developing classification models with noisy annotations is a significant challenge in biomedical applications.
• Existing methods often rely on unreliable annotations from multiple sources.

Purpose of the Study:

• To propose a novel probabilistic classification algorithm for handling noisy annotations from multiple sources.
• To improve the accuracy and reliability of classification models in biomedical contexts.

Main Methods:

• Developed a probabilistic classification algorithm leveraging labels from multiple noisy annotators.
• Implemented a consensus labelling strategy that prioritizes higher-quality annotations.
• Evaluated the algorithm on text classification and protein disorder prediction tasks.

Main Results:

• The proposed algorithm effectively eliminates annotations from novice labellers.
• Achieved more accurate ground truth estimation through consensus labelling.
• Demonstrated superior accuracy, effectiveness, and performance compared to alternative methods in evaluations.

Conclusions:

• The method is suitable for meta-learning from diverse classification models and human-generated noisy annotations.
• Particularly advantageous when dealing with a high proportion of novice labellers.
• Enables annotator characterization to identify competent annotators for specific data instances, leading to more accurate classifiers.