A closer look at how experience, task domain, and self-confidence influence reliance towards algorithms

Area of Science:

• Human-Computer Interaction
• Decision Science
• Machine Learning

Background:

• Prior research indicates algorithm experience reduces reliance behaviors.
• The impact of model experience on reliance intentions remains unexplored.
• Self-confidence and domain knowledge are hypothesized to affect algorithm reliance.

Purpose of the Study:

• To investigate the influence of statistical model experience, task domain, and self-confidence on reliance intentions and behaviors.
• To examine effects on perceived accuracy of personal estimates and model predictions.
• To understand factors driving forecasters' decisions between model predictions and self-estimates.

Main Methods:

• Online study with 347 participants completing a forecasting task.
• Statistical analysis to assess the impact of model experience, task domain, and self-confidence.
• Measured reliance intentions, reliance behaviors, and perceived accuracy.

Main Results:

• Self-confidence and task domain significantly affected reliance intentions, behaviors, and perceived accuracy.
• Model experience did not significantly influence reliance behavior, intentions, or perceived accuracy.
• Task domain (used car sales, GPA, GitHub pull requests) showed significant effects.

Conclusions:

• Self-confidence and task domain are stronger predictors of algorithm reliance than model experience.
• Individual differences and situational factors critically influence forecasting decisions.
• Over-reliance on self-estimates or model predictions can lead to sub-optimal outcomes.