A primer on reliability testing of a rating scale

In this article, the second of a series on rating scale translation, adaptation, and psychometric testing, we focus on reliability testing of a rating scale. Reliability refers to the consistency of results when the scale is reapplied to or completed by the same individual again under the same conditions. We discuss three key types of reliability: internal consistency, test-retest reliability, and inter-rater reliability testing. The appropriate measure for reporting internal consistency is Cronbach's alpha (α); for test-retest reliability, it is the intraclass correlation coefficient (ICC) for continuous variables and intraclass kappa for categorical variables. For inter-rater reliability, the preferred measure is either Cohen's kappa (κ) in case of categorical variables with two raters or the ICC for continuous variables; depending on the randomness in the selection of raters, different statistical models are used for computing the ICC. This article presents these concepts with simple, non-technical explanations. We also address practical considerations for conducting reliability tests, explain how to choose the right statistical index for each type of reliability, and clarify common misapplications. Finally, we offer guidance on interpreting and reporting reliability test results in a manuscript, along with instructions on conducting these analyses using IBM SPSS Statistics.