Evaluation of a human factors analysis and classification system used by trained raters

The U.S. Department of Defense (DoD) has used the DoD Human Factors Analysis and Classification System (DoD-HFACS) to help identify and classify human factors that may have caused or contributed to aircraft mishaps since 2005.

In this study 22 military officers used DoD-HFACS to classify information obtained from an interview with an individual who had been involved in an aviation incident in which the potential for serious injury had been high.

It was found that although the overall interrater reliability was generally acceptable (as reflected by a mean Fleiss' kappa of 0.75) and there were high levels of agreement regarding the factors that did not contribute to the incident (there was agreement of 50% or greater between raters for 84.4% of unselected nanocodes); the level of agreement on the factors that did cause the incident as classified using DoD-HFACS were lower than desirable (agreement of 50% or greater between raters that a particular nanocode was causal was found only for a mean of 22.5% of selected nanocodes).

The findings from this study are consistent with the small number of other studies reporting an evaluation of the reliability of DoD-HFACS. It is recommended that organizations must evaluate the reliability and validity of mishap coding systems, as applied by the proposed end-users, prior to the widespread adoption of a system. It is only through the accurate identification of mishap causal factors that informed decisions can be made to prevent future mishaps.