Management by consent in human-machine systems: when and why it breaks down

This study examined the effects of conflict type, time pressure, and display design on operators' ability to make informed decisions about proposed machine goals and actions in a management-by-consent context. A group of 30 B757 pilots were asked to fly eight descent scenarios while responding to a series of air traffic control clearances. Each scenario presented pilots with a different conflict that arose from either incompatible goals contained in the clearance or inappropriate implementation of the clearance by automated flight deck systems. Pilots were often unable to detect these conflicts, especially under time pressure, and thus failed to disallow or intervene with proposed machine actions. Detection performance was particularly poor for conflicts related to clearance implementation. These conflicts were most likely to be missed when automated systems did more than the pilot expected of them. Performance and verbal protocol data indicate that the observed difficulties can be explained by a combination of poor system feedback and pilots' difficulties with generating expectations of future system behavior. Our results are discussed in terms of their implications for the choice and implementation of automation management strategies in general and, more specifically, with respect to risks involved in envisioned forms of digital air-ground communication in the future aviation system. Actual or potential applications of this research include the design of future data link systems and procedures, as well as the design of future automated systems in any domain that rely on operator consent as a mechanism for human-machine coordination.