Plausible values: how to deal with their limitations

Rasch modeling and plausible values methodology were used to scale and report the results of the Organization for Economic Cooperation and Development's Programme for International Student Achievement (PISA). This article will describe the scaling approach adopted in PISA. In particular it will focus on the use of plausible values, a multiple imputation approach that is now commonly used in large-scale assessment. As with all imputation models the plausible values must be generated using models that are consistent with those used in subsequent data analysis. In the case of PISA the plausible value generation assumes a flat linear regression with all students' background variables collected through the international student questionnaire included as regressors. Further, like most linear models, homoscedasticity and normality of the conditional variance are assumed. This article will explore some of the implications of this approach. First, we will discuss the conditions under which the secondary analyses on variables not included in the model for generating the plausible values might be biased. Secondly, as plausible values were not drawn from a multi-level model, the article will explore the adequacy of the PISA procedures for estimating variance components when the data have a hierarchical structure.