People make mistakes: Obtaining accurate ground truth from continuous annotations of subjective constructs

Area of Science:

• Psychometrics
• Human-Computer Interaction
• Cognitive Science

Background:

• Accurate temporal representation of mental states is vital for understanding complex dynamics.
• Limited methodological research exists on the validity and reliability of continuous-time annotations.
• Current approaches to continuous-time annotation face significant threats to validity and reliability.

Purpose of the Study:

• To present a psychometric perspective on valid and reliable construct assessment over time.
• To examine the robustness of interval-scale continuous-time annotation.
• To propose a novel pipeline for generating robust ground truth data.

Main Methods:

• A psychometric approach was used to assess validity and reliability.
• The robustness of interval-scale continuous-time annotation was examined.
• A novel ground truth generation pipeline combining emerging techniques was developed.

Main Results:

• Three major threats to validity and reliability in current continuous annotation methods were identified.
• The proposed pipeline demonstrated effectiveness in a case study on crowd-sourced movie violence annotation.
• The pipeline achieved a .95 Spearman correlation in summarized ratings, a significant improvement over a .15 baseline.

Conclusions:

• Highly accurate ground truth signals can be generated from continuous annotations.
• Comparative annotation (e.g., a versus b) can correct structured errors in continuous data.
• A paradigm shift towards robust construct measurement over time is necessary.