How Do Test Takers Interact With Simulation-Based Tasks? A Response-Time Perspective.

Area of Science:

• Educational Measurement
• Psychometrics
• Human-Computer Interaction

Background:

• Traditional assessments (multiple-choice, constructed-response) measure fundamental skills.
• Virtual performance assessments (VPAs), including game- or simulation-based tasks, measure integrated skills via interactive behaviors.
• Understanding response times in VPAs is crucial for their development and application.

Purpose of the Study:

• To conduct a systematic timing study of a recently designed simulation-based task.
• To understand response time patterns in complex simulation-based tasks.
• To explore leveraging timing information for designing, assembling, and scoring VPAs.

Main Methods:

• Collected timing data from a simulation-based task developed at Educational Testing Service.
• Performed five complementary analyses to examine statistical properties of timing data.
• Investigated relationships between timing patterns, performance, demographics, motivation, personality, and behaviors using various statistical approaches.

Main Results:

• The five analyses provided complementary insights into test-taker behavioral features within the simulation.
• Timing data revealed useful aspects of participant behavior in the virtual environment.
• Findings were compared with existing literature on timing data in assessments.

Conclusions:

• Response time data in simulation-based assessments offers valuable information beyond traditional metrics.
• Timing patterns can inform the design, assembly, and scoring of complex virtual performance assessments.
• Further research can leverage these findings to enhance the validity and utility of VPAs.