Impact of observers' experience and training on reliability of observations for a manual handling task

Observation reliability (agreement percentage and kappa coefficients) for six experienced ergonomists and six untrained participants was computed. Participants were first tested after a training session and 1 week later after an additional practice session. Two formats were used: free practice and directed exercise. Reliability was tested for 17 variables and 20 sequences using photographic and video supports. The participants were asked to indicate whether they were confident about their answer, to rate this confidence on a scale of 1 to 10, and when the confidence rating was below 8, to provide a reason for this. Experience and additional practices had no clear impact on reliability, which was excellent overall. The main reason given was that the event to be observed took place at the borderline between two classes. The observers' rating on the scale appeared to be tied to the subsequent reliability computed. The use of a confidence scale appeared to be a useful tool for forecasting observation problems.