Evaluating user interfaces in context: the ecological value of time-and-motion studies

Findings are summarized from a study which set out to identify cognitive stumbling blocks in the user interface of a large computer system used by telephone operators in Australia. Intermittent observations were made of operators' actions in the workplace over a period of eight months before, during and after system implementation. Numerous weaknesses were identified in the user interface, but the most interesting aspect of the study turned out to be the analysis of the overall jobs operators are required to do, as a by-product of the intended study. The insight into operator job demands led to changes in job selection criteria and training, which were found to match actual job demands quite poorly at the time the study was conducted. The paper describes the process by which the usability assessment of the user-system interface led to a comprehensive job analysis. Supporting quantitative data are presented, together with anecdotal examples which demonstrate the importance of conducting systems analysis in the working context, thereby maximizing the ecological value of the resulting research data.