A Time-Series-Based New Behavior Trace Model for Crowd Workers That Ensures Quality Annotation

Area of Science:

• Computer Science
• Human-Computer Interaction

Background:

• Crowdsourcing leverages internet users for tasks but faces quality issues due to diverse worker backgrounds.
• Existing quality control methods include consensus and gold standards, but worker behavior tracing is preferred.
• Analyzing worker behavior is crucial for enhancing crowdsourcing task classification and quality assurance.

Purpose of the Study:

• To propose two novel models for crowdsourcing task classification based on worker behavior.
• To incorporate time-series features and characteristics for improved model performance.
• To evaluate the effectiveness of the proposed models against state-of-the-art baselines.

Main Methods:

• Developed two models leveraging worker behavior for task classification.
• Model 1: Utilized multiple time-series features with a machine learning classifier.
• Model 2: Converted time-series data into images using recurrent characteristics and applied a convolutional neural network (CNN).

Main Results:

• The feature-based model achieved an accuracy of 83.8%.
• The CNN model achieved an accuracy of 76.6%.
• Both proposed models demonstrated superior performance compared to current state-of-the-art methods.

Conclusions:

• Worker behavior analysis, particularly using time-series features, offers a promising approach for crowdsourcing task classification.
• The proposed feature-based and CNN models effectively address quality concerns in crowdsourcing.
• These novel models represent a significant advancement in the field of crowdsourcing quality management.