Psychometric Modeling to Identify Examinees' Strategy Differences during Testing.

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Psychology

Background:

• Standard interpretation of aptitude test scores assumes uniform examinee strategies.
• Research indicates significant individual differences in test-taking strategies and procedural adherence.
• These strategy variations can compromise the validity of score interpretations for examinees with similar overall scores.

Purpose of the Study:

• To present item response theory (IRT) models for identifying examinee strategy differences.
• To explore psychometric approaches utilizing response time data to analyze response processes.
• To propose a general method for evaluating threats to the validity of psychological measurement instruments.

Main Methods:

• Application of mixture IRT models to identify latent classes of examinees with distinct item response patterns.
• Modeling response patterns within identified classes using cognitive complexity variables to study strategy differences.
• Utilizing novel psychometric approaches with response time residuals to capture inter- and intraindividual variability in response processes.

Main Results:

• Demonstrated the utility of mixture IRT models in distinguishing examinee groups based on response patterns.
• Showcased the effectiveness of response time residuals in identifying variability in cognitive processes during testing.
• Proposed and validated a method for assessing threats to measurement validity using these advanced techniques.

Conclusions:

• Advanced IRT models and response time analysis offer powerful tools for uncovering examinee strategy differences.
• These methods enhance the interpretability of performance estimates and improve psychological measurement instruments.
• The proposed approach contributes to more nuanced and valid interpretations of aptitude test scores.