Measures of operator performance in complex, dynamic microworlds: advancing the state of the art

Microworld research provides a useful complement to field studies and highly controlled laboratory studies, aiming to strike a balance between representativeness and experimental control. Yet microworld research has associated methodological difficulties, particularly the problem of performance measurement. Researchers generally adopt a variety of measures to provide converging evidence concerning questions of interest. To confront problems with existing measures, this paper examines a series of objective measures used to characterize the performance of human operators in process control. These measures include novel, quantitative extensions to existing graphical analyses and new graphical representations. The measures are applied in the context of a 6-month longitudinal study using an interactive, thermal-hydraulic process control microworld (DURESS II). The following measures are discussed: steady-state time, action transition graph complexity, the path length in state space diagrams, the area under distance-to-goals graphs, divergence from the temperature goal line in mass inventory versus energy inventory graphs, and the proportion of control actions near the beginning of the trials represented by timelines. Two case studies emphasize the performance and strategy differences of individual operators across the battery of measures.