Evaluation of a Human Factors Analysis and Classification System as used by simulated mishap boards

The reliability of the Department of Defense Human Factors Analysis and Classification System (DOD-HFACS) has been examined when used by individuals working alone to classify the causes of summary, or partial, information about a mishap. However, following an actual mishap a team of investigators would work together to gather and analyze a large amount of information before identifying the causal factors and coding them with DOD-HFACS.

There were 204 military Aviation Safety Officer students who were divided into 30 groups. Each group was provided with evidence collected from one of two military aviation mishaps. DOD-HFACS was used to classify the mishap causal factors.

Averaged across the two mishaps, acceptable levels of reliability were only achieved for 56.9% of nanocodes. There were high levels of agreement regarding the factors that did not contribute to the incident (a mean agreement of 50% or greater between groups for 91.0% of unselected nanocodes); the level of agreement on the factors that did cause the incident as classified using DOD-HFACS were low (a mean agreement of 50% or greater between the groups for 14.6% of selected nanocodes).

Despite using teams to carry out the classification, the findings from this study are consistent with other studies of DOD-HFACS reliability with individuals. It is suggested that in addition to simplifying DOD-HFACS itself, consideration should be given to involving a human factors/organizational psychologist in mishap investigations to ensure the human factors issues are identified and classified in a consistent and reliable manner.