Health and safety management system audit reliability pilot project

This pilot study assessed occupational health and safety (OHS) management system audit finding reliability using a modified test-retest method. Two industrial hygienists with similar training and education conducted four, 1-day management system audits in four dissimilar organizational environments. The researchers examined four auditable sections (employee participation, training, controls, and communications) contained in a publicly available OHS management system assessment instrument. At each site, 102 auditable clauses were evaluated using a progressive 6-point scale. The team examined both the consistency of and agreement between the scores of the two auditors. Consistency was evaluated by calculating the Pearson r correlations for the two auditors' scores at each site and for each section within each site. Pearson correlations comparing overall scores for each site were all very low, ranging from 0.206 to 0.543. Training and communication system assessments correlated the highest, whereas employee participation and control system scores correlated the least. To measure agreement, t-tests were first calculated to determine whether the differences were statistically significant. Aggregate mean scores for two of the four sites were significantly different. Of the 16 total sections evaluated (i.e., 4 sections per site), seven scores were significantly different. Finally, the agreement of the scores between the two auditors for the four sites was evaluated by calculating two types of intraclass correlation coefficients, all of which failed to meet the minimum requirement for agreement. These findings suggest that opportunities for improving the reliability of the instrument and the audit process exist. Future research should include governmental and commercial OHS program assessments and related environmental management systems and their attendant audit protocols.