How accurate are subjective judgements of a continuum?

Area of Science:

• Psychophysics
• Perception Science
• Statistical Modeling

Background:

• Subjective judgments are crucial in many scientific fields.
• Assessing the accuracy of these judgments on continua is vital for reliable data collection.

Purpose of the Study:

• To evaluate the precision of subjective estimations on a uni-dimensional continuum.
• To determine the impact of scale granularity and observer pooling on accuracy.

Main Methods:

• Twenty-four observers estimated dot positions on a 100-point scale.
• Data analyzed for deviation from true values, correlation, kappa agreement, and information transmission.

Main Results:

• Ninety-five percent of estimates fell within +/- 9 of the true value.
• Median and mean estimates closely approximated true values (max deviation of 5).
• High correlation (r=+1) between ranked medians/means and true values; kappa varied with categories.

Conclusions:

• Subjective judgments can be highly accurate with numerous categories.
• Pooling observer data refines confidence intervals and approaches objective reality.
• Findings suggest a potential gold standard for ranking pathological continua and highlight limitations of kappa statistics for continua.