Evaluation of the HFACS-ADF safety classification system: inter-coder consensus and intra-coder consistency

This article evaluates an adaptation of the human factors analysis and classification system (HFACS) adopted by the Australian Defence Force (ADF) to classify factors that contribute to incidents. Three field studies were undertaken to assess the reliability of HFACS-ADF in the context of a particular ADF air traffic control (ATC) unit. Study one was designed to assess inter-coder consensus between many coders for two incident reports. Study two was designed to assess inter-coder consensus between one participant and the previous original analysts for a large set of incident reports. Study three was designed to test intra-coder consistency for four participants over many months. For all studies, agreement was low at the level of both fine-level HFACS-ADF descriptors and high-level HFACS-type categories. A survey of participants suggested that they were not confident that HFACS-ADF could be used consistently. The three field studies reported suggest that the ADF adaptation of HFACS is unreliable for incident analysis at the ATC unit level, and may therefore be invalid in this context. Several reasons for the results are proposed, associated with the underlying HFACS model and categories, the HFACS-ADF adaptations, the context of use, and the conduct of the studies.