Exploring the Multiverse of Analytical Decisions in Scaling Educational Large-Scale Assessment Data: A Specification...

Area of Science:

• Educational Measurement
• Psychometrics
• Statistics

Background:

• Large-scale assessment (LSA) studies like PISA use item response theory (IRT) to analyze student performance across countries.
• Model specification choices in IRT can influence the interpretation of results.

Purpose of the Study:

• To investigate the impact of various model specification factors on country-level parameters in the PISA 2018 mathematics assessment.
• To evaluate the extent of model uncertainty in large-scale assessment scaling.

Main Methods:

• A multiverse analysis was conducted, examining five key factors in the PISA scaling model: IRT model form, differential item functioning, missing data treatment, item selection, and test position effects.
• The analysis focused on obtaining country means and standard deviations.

Main Results:

• Model uncertainty was found to have a substantial impact on country means, comparable to sampling errors.
• Model uncertainty's effect on country standard deviations was even larger than standard errors.
• Each of the five investigated specification factors demonstrated at least a moderate effect on country means or standard deviations.

Conclusions:

• Model specification decisions in LSA studies warrant critical evaluation.
• It is recommended to either report the variability stemming from model uncertainty or select the most valid model specification.
• Model fit should not be the primary criterion for selecting scaling strategies in LSA.