IRT Scoring and Test Blueprint Fidelity

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Test specifications, or blueprints, guide assessment design and content allocation.
• Item Response Theory (IRT) provides statistical models for test scoring.
• Standard IRT models use optimal scoring weights derived from item parameters.

Purpose of the Study:

• To investigate whether standard IRT scoring models accurately reflect the intended content allocation specified in a test blueprint.
• To assess if the scoring weights generated by IRT models align with the proportion of items designated for each content area in the blueprint.
• To specifically examine the implications of the three-parameter logistic (3PL) model, where scoring weights are ability-dependent, on intended content representation.

Main Methods:

• Analysis of scoring weights derived from standard Item Response Theory (IRT) models, specifically the two-parameter logistic (2PL) and three-parameter logistic (3PL) models.
• Comparison of IRT-generated optimal scoring weights against a predefined set of intended weights based on the proportion of items in each cell of a test blueprint.
• Focus on the 3PL model to evaluate how ability-dependent scoring weights might influence the representation of content areas.

Main Results:

• Standard IRT scoring models employ optimal weights that are contingent upon item parameters.
• The three-parameter logistic (3PL) model's scoring weights are influenced by examinee ability levels.
• A potential discrepancy exists where the intended content weights, as defined by the test blueprint, may be implicitly altered for examinees with lower ability in the 3PL model.

Conclusions:

• The alignment between IRT scoring models and test blueprint content allocation requires careful consideration, particularly with complex models like the 3PL.
• The ability-dependent nature of scoring weights in the 3PL model raises concerns about maintaining intended content representation across all examinee ability levels.
• Further research is needed to ensure IRT scoring accurately reflects test design specifications, especially for diverse examinee populations.