Modeling Booklet Effects for Nonequivalent Group Designs in Large-Scale Assessment

Area of Science:

• Educational Measurement
• Psychometrics
• Statistics

Background:

• Large-scale assessments commonly use matrix designs with multiple booklets to distribute test items.
• While reducing student burden and aligning difficulty, multiple booklets may introduce unwanted response variation.

Purpose of the Study:

• To provide guidance on identifying and modeling booklet effects in large-scale assessments.
• To investigate the presence and impact of booklet effects in a national educational standards assessment.

Main Methods:

• Developed three statistical models based on the Rasch model using generalized linear mixed models.
• Applied these models to data from a national assessment of scientific competence involving 9,044 students in Grades 9 and 10.
• Utilized 1,021 items distributed across 74 booklets.

Main Results:

• A small but statistically significant booklet effect was detected.
• The results indicate that booklet design can influence response behavior in large-scale assessments.

Conclusions:

• Booklet effects, though small, are a non-negligible factor in large-scale assessments.
• It is recommended to assess and account for booklet effects in subsequent analyses to ensure accurate measurement.