Analysis of paired Likert data: how to evaluate change and preference questions

Likert scale data present unique analysis concerns that are often not recognized by nonstatistical researchers. Mean responses to a single question are frequently compared among two or more groups of moderate sample size using normal theory methods (t test, ANOVA). These tests are designed for continuous normally distributed data, but Likert responses are categorical, ordinal, and not normally distributed. The problems are further complicated when the paired nature of pre/post- or preference questioning is ignored.

Simulated data for a single five-point Likert scale question was used to illustrate the differing conclusions that may arise from single-question Likert scale data depending on whether pairing is modeled appropriately and which statistical procedure was applied (two sample: t test or Wilcoxon rank sum; paired samples; t test, Wilcoxon signed rank, or sign test).

Cases are shown for which the paired responses are significantly different (indicated by the sign test), although the group mean responses to the two questions are identical (leading to nonsignificance with both paired t and signed rank tests). The sign test is shown to be appropriate and simple to apply in typical pre/post- or preference situations.

Correctly identifying the question of interest is critical to selecting the appropriate statistical methodology.