Enhancing Content Validity Assessment With Item Response Theory Modeling.

Area of Science:

• Psychometrics
• Educational Measurement
• Assessment Validity

Background:

• Ensuring assessment validity relies on rigorous test content evaluation.
• Subject matter experts (SMEs) commonly assess item relevance, representativeness, and appropriateness.
• This study introduces item response theory (IRT) for SME model assessments.

Purpose of the Study:

• To integrate IRT into SME assessments for evaluating item relevance and SME performance.
• To compare IRT-based validity with traditional indices like content validity index and Aiken's V.
• To assess SME accuracy in identifying trait-specific items and predicting factor loadings.

Main Methods:

• Item response theory (IRT) was applied to model SME evaluations.
• IRT-derived SME parameters (discrimination, threshold) were estimated.
• IRT scores were compared against content validity index and Aiken's V for predictive accuracy.

Main Results:

• IRT scores accurately identified conscientiousness items (R2 = 0.57) and predicted factor loadings (R2 = 0.45).
• IRT demonstrated incremental validity, explaining 11-17% more variance than traditional indices.
• IRT effectively detected suboptimal SME performance in item evaluation.

Conclusions:

• Modeling SME assessments with IRT enhances item alignment with intended constructs.
• IRT provides improved prediction of factor loadings, strengthening content validity.
• This approach facilitates the development of more valid measurement instruments.