Testing the reliability of accident analysis methods: a comparison of AcciMap, STAMP-CAST and AcciNet

Area of Science:

• Safety Science
• Human Factors Engineering
• Systems Engineering

Background:

• Accident analysis methods are vital for understanding complex incidents.
• Systemic approaches like AcciMap, STAMP-CAST, and AcciNet identify safety interventions across sociotechnical systems.
• The reliability of these systemic methods for classifying causal factors is not well-established.

Purpose of the Study:

• To assess the intra-rater and inter-rater reliability of AcciMap, STAMP-CAST, and AcciNet.
• To compare the reliability of these three systemic accident analysis methods.
• To apply the Signal Detection Theory (SDT) paradigm for reliability assessment.

Main Methods:

• An intra-rater and inter-rater reliability assessment was conducted.
• Three systemic accident analysis methods (AcciMap, STAMP-CAST, AcciNet) were evaluated.
• Thirty expert analysts performed 360 comparisons over 180 hours, analyzed using Signal Detection Theory.

Main Results:

• All three methods showed weak to moderate positive correlation coefficients.
• STAMP-CAST exhibited significantly higher inter-rater reliability than AcciMap and AcciNet.
• No significant differences in intra-rater reliability were observed across the methods.

Conclusions:

• While all methods have moderate reliability, STAMP-CAST offers more consistent results between different analysts.
• Findings highlight the importance of inter-rater reliability for systemic accident analysis methods.
• Further research is needed to explore implications for safety intervention identification.