Experiment-based calibration: Inference and decision-making

Experiment-based calibration is an emerging approach for measurement validation in the behavioural sciences. It allows comparing multiple measurement methods by how well they reproduce a known experimental manipulation, providing insight into their measurement accuracy. Calibration entails questions unparalleled in classical validation approaches. The first is about inference: when should we conclude that one measurement method is truly more accurate than another? The second is about decisions: when should we decide that a method merits the investment of changing a measurement system? In this note, we review these questions in the context of the statistical challenges that arise in a calibration process: a potentially large and a priori unknown number of measurement methods; a requirement to integrate evidence across multiple calibration samples; and a possibility that some methods may not be available for all samples. We show that Bayesian meta-analytic model comparison is a suitable framework for inference in calibration, and propose a decision-theoretic approach to calculate immediate economic gain garnered through reduced sample sizes. In order to overcome the practical hurdles associated with the analysis of calibration experiments, we present CalibR, an R package for calibration inference.