Using Sankey diagrams to visualize drag and drop action sequences in technology-enhanced items

Area of Science:

• Educational Measurement
• Data Visualization
• Cognitive Psychology

Background:

• Traditional assessment analysis often overlooks detailed student interaction processes.
• Existing methods for analyzing technology-enhanced item data are limited, especially for complex actions like drag-and-drop.
• Understanding the step-by-step actions of test-takers is crucial for deeper insights into problem-solving.

Purpose of the Study:

• To propose a novel Sankey diagram visualization method for analyzing test-takers' action sequences in drag-and-drop items.
• To extract comprehensive, item-level process data from test-taker interactions.
• To explore the relationship between domain knowledge, response strategies, and item performance.

Main Methods:

• Development of a Sankey diagram visualization technique tailored for drag-and-drop interactions.
• Extraction and analysis of process data from mathematical items in the National Assessment of Educational Progress (NAEP).
• Identification of frequent actions, deduction of response strategies, and examination of strategy-performance links.

Main Results:

• Demonstrated the feasibility of using Sankey diagrams to visualize complex action sequences.
• Identified distinct patterns of actions and common problem-solving strategies employed by test-takers.
• Revealed correlations between specific strategies, mathematical knowledge, and overall item performance.

Conclusions:

• The Sankey diagram method offers a powerful tool for process-based analysis in educational assessment.
• This approach enhances the understanding of how students engage with and solve problems in technology-enhanced environments.
• Future research can extend this method to various assessment types and further investigate cognitive processes during testing.