Related Experiment Videos
On Setting Response Criteria for Calibrated Subjective Probability Estimates.
Hongbin Gu1, Thomas S. Wallsten
1University of North Carolina at Chapel Hill
Journal of Mathematical Psychology
|August 9, 2001
Summary
This study introduces a new algorithm for finding well-calibrated response criteria in signal-detection theory. Perfect calibration is achievable for moderately difficult tasks but not for very easy or very hard ones.
Area of Science:
- Cognitive Psychology
- Decision Science
- Signal Detection Theory
Background:
- Subjective probability calibration is crucial for accurate decision-making.
- Previous research has not fully addressed the conditions for optimal calibration.
Purpose of the Study:
- To develop an algorithm for identifying well-calibrated response criteria.
- To investigate the relationship between task difficulty and the attainability of perfect calibration.
Main Methods:
- The study frames subjective probability calibration within signal-detection theory.
- A novel algorithm is developed and proven to find well-calibrated criteria.
- The algorithm is applied to tasks with varying difficulty and response categories.
Main Results:
- Perfect calibration is most attainable at median task difficulty levels (d' ≈ 1.4).
- Attaining perfect calibration is practically impossible for very hard (d' ≈ 0.5) or very easy (d' ≈ 10) tasks.
- This finding supports the 'hard-easy effect' in calibration research.
Conclusions:
- The developed algorithm reliably identifies well-calibrated response criteria.
- Task difficulty significantly impacts the possibility of achieving perfect subjective probability calibration.
- Findings have implications for understanding and improving calibration in various research and applied settings.