Assessing the Unconditional and Conditional External Validity of Noncognitive Test Scores: A Unifying Model-Based...

Area of Science:

• Psychometrics
• Statistical Modeling

Background:

• External validity evidence for score estimates remains crucial in psychometrics.
• Model-based approaches to external validity, particularly integrating SEM and IRT, have been underexplored.
• Current Item Response Theory (IRT) often prioritizes internal properties over external score validity.

Purpose of the Study:

• To develop and propose a novel model-based approach for assessing the external validity of score estimates in noncognitive measures.
• To integrate advancements from Structural Equation Modeling (SEM) and IRT for a comprehensive external validity assessment.
• To provide practical tools for evaluating and enhancing the external validity of psychometric measures.

Main Methods:

• Development of a general extended model incorporating external variables.
• Derivation and structural fitting of four well-known extended IRT models from the general model.
• Proposal of unconditional and conditional indices to quantify the relationship between score estimates and external variables.

Main Results:

• A framework is established for detailed external validity assessment of score estimates.
• The proposed indices offer insights into population-dependent and population-independent relationships.
• Demonstration of practical applications including model appropriateness, individual prediction, and test optimization.

Conclusions:

• The proposed SEM-IR T integration offers a robust method for assessing external validity in noncognitive measures.
• The developed indices and framework support practical psychometric applications, enhancing score interpretation and test development.
• The approach provides a valuable tool for researchers and practitioners seeking to validate psychometric measures effectively.