Reliability and Validity of Six Selected Observational Methods for Risk Assessment of Hand Intensive and Repetitive...

Area of Science:

• Occupational health and safety
• Ergonomics
• Risk assessment methodology

Background:

• Observational methods are standard for assessing risks in hand-intensive and repetitive work.
• Comparisons of these methods are challenging due to variations in observer expertise, task complexity, and statistical approaches.

Purpose of the Study:

• To evaluate the inter- and intra-observer reliability and concurrent validity of six distinct risk assessment methods.
• To utilize a standardized methodological design and statistical analysis across all evaluated methods for robust comparison.

Main Methods:

• Twelve experienced ergonomists assessed ten video-recorded work tasks twice using six different methods.
• Concurrent validity was established through consensus assessments by three independent experts.
• Inter- and intra-observer reliability and concurrent validity were quantified using linearly weighted kappa values.

Main Results:

• All methods demonstrated inter-observer reliability (total-risk kappa) below 0.5 (0.15-0.45), indicating agreement below 50% after accounting for chance.
• Concurrent validity values were also moderate, ranging from 0.31 to 0.54 (total-risk kappa).
• Intra-observer reliability showed only a slight improvement (0.16-0.58), with specific methods like ART and HARM being sensitive to work task duration.

Conclusions:

• Systematic observational risk assessment methods, even when used by experienced ergonomists, exhibit low reliability and moderate validity.
• Difficulties in rating hand/wrist postures were noted, consistent with previous research.
• Complementing observational methods with technical assessments is recommended for evaluating ergonomic interventions, especially given the substantial risk of misclassification.