Effects of Design Properties on Parameter Estimation in Large-Scale Assessments

Area of Science:

• Educational Measurement
• Psychometrics
• Large-Scale Assessment Design

Background:

• Booklet design is crucial for accurate student achievement assessment.
• Optimizing item position balance and item pair co-occurrence in booklets is common practice.

Purpose of the Study:

• To investigate the impact of position balance and cluster pair balance on item parameter bias and root mean square error (RMSE) within the Rasch model.
• To provide evidence-based recommendations for optimizing large-scale assessment booklet design.

Main Methods:

• Estimating position effects using data from a large-scale science assessment of 19,107 ninth graders.
• Conducting a simulation study with 1,540 booklet designs varying position and cluster pair balance.
• Analyzing bias and RMSE of item parameter estimates using the Rasch model.

Main Results:

• Position balancing showed a small but significant effect on reducing bias and RMSE of item parameter estimates.
• Cluster pair balance demonstrated an ignorable effect on item parameter estimation accuracy.
• The findings suggest that reducing cluster pair balance does not negatively impact item parameter estimates.

Conclusions:

• Achieving high position balance is recommended for large-scale assessment design to improve parameter estimation.
• Test designers can reduce the emphasis on cluster pair balance without compromising the accuracy of item parameter estimates.
• These findings offer practical guidance for efficient and effective large-scale assessment development.