Controlling Rater Effects in Divergent Thinking Assessment: An Item Response Theory Approach to Individual Response...

Area of Science:

• Psychometrics
• Cognitive Psychology

Background:

• Scoring divergent thinking (DT) tasks is challenging due to inter-rater variability.
• Item Response Theory (IRT) provides a statistical framework to address rater severity and discrimination differences.

Purpose of the Study:

• To apply and evaluate the Item Response Theory (IRT) framework for scoring divergent thinking (DT) tasks.
• To compare IRT-adjusted scores with traditional scoring methods regarding reliability and potential confounding effects.
• To assess the robustness of IRT approaches with simulated missing data.

Main Methods:

• Re-analysis of an open-access dataset of 202 participants' DT task scores.
• Comparison of various IRT models, selecting the Graded Response Model as the best fit.
• Examination of rater severity and discrimination parameters for individual response and snapshot scoring.
• Simulation of missing data to test approach robustness.

Main Results:

• IRT models are applicable to both individual response scoring and snapshot scoring of DT tasks.
• IRT-adjusted and unadjusted scores demonstrated high correlations, especially under high inter-rater agreement.
• Rater variability in severity and discrimination had minimal impact on scores when agreement was high.

Conclusions:

• Item Response Theory (IRT) is a valuable tool for modeling rater severity and discrimination in DT scoring.
• IRT offers practical benefits for DT scoring, though specific conditions for optimal application require further investigation.
• The study confirms IRT's utility in psychometric analysis of complex cognitive tasks.