Stability of questionnaire items in sport and exercise psychology: bootstrap limits of agreement

We describe an alternative method for assessing the stability of individual questionnaire items in sport and exercise psychology. To date, the Pearson product-moment correlation coefficient has been widely used in psychometrics. We propose an alternative non-parametric method based on proportion of agreement. Ninety-two male university students completed the revised 9-item Social Physique Anxiety Scale on two occasions, separated by a 2-week interval. Point estimates of the proportion of direct within-individual agreement between the two occasions were calculated separately for each item of the Social Physique Anxiety Scale. Estimates of uncertainty of the agreement were calculated using a bootstrapping resampling technique. For each item, 2000 bootstrap samples (each n = 92 pairs) were redrawn from the original sample. The sample statistic was calculated for each bootstrap sample to provide a bootstrap sampling distribution. The 95% confidence intervals (CI) were then calculated using the percentile method. The three most problematic items were items 7, 8 and 10 (as labelled in the original 12-item scale). These items demonstrated an agreement of 0.46 (95% CI= 0.36 0.56), 0.42 (95% CI = 0.33-0.52) and 0.41 (95% CI = 0.32-0.51) respectively. Our proposed method measures absolute agreement between test-retest responses, is free of normal assumptions, does not depend on high between-individuals variance, and can be applied successfully to individual items in the development of psychological tests.