Discriminative Cut-Offs, Concurrent Criterion Validity, and Test-Retest Reliability of the Oxford Vaccine Hesitancy...

Validated instruments can be used to quantify vaccine hesitancy, but few provide transportable operational cut-offs or temporal stability metrics. We evaluated the Oxford Vaccine Hesitancy Scale (OVHS) to quantify discrimination for the self-reported historical COVID-19 vaccine refusal, derive and validate a single operational cut-off across populations, and assess the test-retest reliability.

Five datasets (including one from a repeat administration) comprising 2451 assessments from 1989 demographically representative UK and US respondents were analyzed without pooling. Receiver operating characteristic (ROC) curves with 10,000 bootstrap replications (bias-corrected and accelerated and percentile Cis) were used to quantify discrimination. A Youden's J near-optimal plateau algorithm constrained by a cross-dataset specificity floor (≥0.75) was used to select a transportable cut-off. A prevalence-agnostic aggregate Index of Union (Iuagg) provided secondary confirmation of this cut-off. Cut-off behaviour was visualized with a multi-sample two-graph ROC plot. The six-week test-retest reliability on a UK sample used a two-way mixed-effects, absolute-agreement ICC(A,1).

AUCs ranged 0.760-0.971 across datasets (derivation AUC: 0.960). The Youden plateau spanned scores 34-38; applying the specificity floor yielded an operational cut-off OVHS ≥ 35, which was confirmed by the Iuagg. At ≥35, sensitivity/specificity were 0.73-0.95/0.63-0.87 across samples; negative predictive value exceeded 0.90 in all cohorts. The test-retest reliability was good to excellent, with the OVHS total ICC(A,1) = 0.884 (95% CI 0.863-0.903); subscales ranged from 0.649 to 0.901 and the average-measure ICC(A,2) = 0.939.

The OVHS demonstrates good-to-excellent discrimination for historical COVID-19 vaccine refusal and strong temporal stability. We found a single, transparent, and transportable operational cut-off (≥35). Our cut-off derivation framework is scale- and endpoint-agnostic and may generalize to other hesitancy instruments and decision cut-offs more broadly.