Rasch Measurement of Collaborative Problem Solving in an Online Environment

Area of Science:

• Educational Psychology
• Human-Computer Interaction
• Cognitive Science

Background:

• Assessing collaborative problem-solving (CPS) is crucial for understanding complex cognitive skills.
• Online environments offer scalable platforms for CPS assessment.
• Identifying reliable behavioral indicators of CPS is an ongoing research challenge.

Purpose of the Study:

• To develop and validate an approach for assessing human-to-human collaborative problem-solving in an online setting.
• To investigate the impact of self-selected roles (A or B) on individual student performance.
• To identify and validate behavioral indicators of CPS using process stream data.

Main Methods:

• Utilized online interactive tasks with 3402 students across six countries.
• Captured process stream data (clicks, drags, chat logs) to identify behavioral indicators.
• Applied the Rasch simple logistic model to estimate student ability and item difficulty, ensuring cross-cultural invariance.

Main Results:

• Identified behavioral indicators of CPS that demonstrated consistency across national samples, languages, curricula, and ages.
• The Rasch model provided reliable estimates of student ability based on demonstrated behaviors.
• Found no significant effect of role selection on individual student performance measures.

Conclusions:

• Online assessment of human-to-human collaborative problem-solving is feasible using behavioral indicators.
• The developed approach offers a valid and reliable method for measuring CPS in digital environments.
• Behavioral indicators derived from process data can effectively estimate student ability in collaborative tasks.