Improving Crowdsourcing-Based Image Classification Through Expanded Input Elicitation and Machine Learning.

Area of Science:

• Computer Science
• Machine Learning
• Computer Vision

Background:

• Image classification tasks require accurate labels, often obtained through human annotation.
• Crowdsourcing offers a scalable approach to data labeling but can suffer from noise and variability.

Purpose of the Study:

• To investigate how diverse crowdsourcing input elicitation methods improve inferred labels for image classification.
• To evaluate the effectiveness of various voting and machine learning (ML) methods in leveraging these inputs.
• To develop a systematic synthetic image generation process for assessing performance on tasks of varying difficulty.

Main Methods:

• Five input elicitation methods were tested: binary classification, object coordinate, confidence level, perceived majority classification, and task difficulty.
• Two crowdsourcing studies involving over 300 participants were conducted.
• A synthetic image generation process using the MPEG-7 Core Experiment CE-Shape-1 Test Set was developed to create images of varying difficulty.

Main Results:

• Combining crowdsourced binary classification labels with average self-reported confidence values improved ML classifier accuracy, particularly with smaller training datasets.
• Augmenting ML algorithms with automated classifier outputs can yield higher performance than individual classifiers when a larger annotated dataset is available.
• Specific modifications to aggregation methods allow prioritization of performance metrics like reduced false-negative rates.

Conclusions:

• Leveraging multiple crowdsourced data types, especially confidence scores, enhances image classification accuracy.
• Hybrid approaches combining human and automated intelligence offer superior performance in image classification.
• Crowdsourcing methodologies can be adapted to optimize for specific performance metrics beyond simple accuracy.