Crowdsourcing with the drift diffusion model of decision making.

Area of Science:

• Machine Learning
• Computational Neuroscience
• Data Science

Background:

• Crowdsourcing relies on uncertain annotator labels to determine ground truth.
• Estimating annotator reliability (e.g., sensitivity, specificity) is crucial.
• Existing methods often use beta or Dirichlet distributions for annotator reliability priors.

Purpose of the Study:

• To investigate the drift-diffusion model, a neuroscientifically validated decision-making model, as a prior for annotator reliability in crowdsourcing.
• To compare this novel approach against state-of-the-art methods.

Main Methods:

• Applied the drift-diffusion model as a prior on annotator labeling processes.
• Conducted experiments on synthetic data with non-linear decision boundaries.
• Utilized variational inference for predicting ground truth labels and annotator parameters.

Main Results:

• The proposed method demonstrated performance comparable to the state-of-the-art Support Vector Gaussian Process Regression (SVGPCR).
• Achieved similar predictive accuracy for crowdsourced data labels.
• Successfully predicted labels using a crowdsourced Gaussian process classifier.

Conclusions:

• The drift-diffusion model offers a neuroscientifically grounded alternative for modeling annotator behavior in crowdsourcing.
• This approach enhances the understanding of annotator decision-making.
• Opens possibilities for predicting neuroscientific biomarkers of annotators, expanding crowdsourcing insights.