Computational Psychometrics for the Measurement of Collaborative Problem Solving Skills

Area of Science:

• Educational Psychology
• Computational Psychometrics
• Human-Computer Interaction

Background:

• Measuring collaborative problem solving (CPS) skills is challenging.
• Traditional psychometric methods may not capture dynamic, interactive skill use.
• Emerging computational approaches offer new possibilities for skill assessment.

Purpose of the Study:

• To develop and validate a psychometrically-based approach for measuring CPS skills.
• To utilize behavioral data from an online simulation for real-time and post-game skill analysis.
• To apply computational psychometrics techniques to educational data mining.

Main Methods:

• Collected behavioral data from middle school students interacting with a collaborative problem-solving online simulation.
• Employed Evidence-Centered Design (ECD) principles for task development.
• Applied real-time Bayesian evidence tracing and post-game cluster analysis of sub-skill performance.

Main Results:

• Demonstrated a continuous Bayesian evidence tracing method for updating sub-skill mastery probabilities in real-time.
• Identified unique gameplay profiles through cluster analysis of user dialog response patterns.
• Validated the use of computational psychometrics for assessing complex cognitive skills.

Conclusions:

• The computational psychometrics approach effectively measures collaborative problem solving skills.
• Real-time and post-game analyses provide valuable insights into skill acquisition and user behavior.
• This methodology offers a robust framework for assessing complex skills in digital environments.