Using Explanatory Item Response Theory to Study Rating-Scale Design Effects on Response Style Discrimination

This paper develops an explanatory Item Response Theory (IRT) methodology that supports the study of rating-scale design features on item sensitivity to response style. Using item response data from a previous two-part experimental study that systematically manipulated various rating scale features (e.g., the number of rating scale categories, rating scale labeling, and polarity of rating scales), a multidimensional nominal response model (MNRM) methodology is applied in which response style is represented by a latent trait that distinguishes respondents having preferences toward extreme versus middle response categories. Item discrimination parameters on the response style dimension are treated as outcomes within an explanatory IRT framework where the manipulated design features serve as predictors. Results from our analyses suggest that the number of rating scale categories was generally associated with stronger response-style discrimination, while wave, labeling and polarity conditions appeared less influential. Importantly, residual variability in response style discrimination at both the within- and between-item levels was seen in both studies to be quite high, making even statistically significant design features appear relatively weak in their effects. In both studies, the measured construct is seen to play a significant role in explaining some of that between-item heterogeneity. The results point to the feasibility of using the MNRM within an explanatory IRT framework as a basis for exploring item-level sensitivities to response style. The framework also uniquely allows for evaluating design effects against residual within-item and between-item sources of heterogeneity in response style item discrimination. The presence of substantial within- and between-item residual heterogeneity in the explanatory IRT analysis helps explain the frequent inconsistencies seen across regarding the effects of design characteristics.