Observational teamwork assessment for surgery: construct validation with expert versus novice raters.

Area of Science:

• Surgical safety and teamwork assessment.
• Human factors in healthcare.
• Medical education and training.

Background:

• Poor teamwork in surgical settings is linked to adverse patient events.
• The Observational Teamwork Assessment for Surgery (OTAS) tool evaluates real-time teamwork in surgical teams.
• Limited research exists on how expert versus novice users apply the OTAS tool.

Purpose of the Study:

• To evaluate the construct validity of the Observational Teamwork Assessment for Surgery (OTAS) tool.
• To compare scoring consistency between expert/expert and expert/novice rater pairs using the OTAS.
• To identify differences in teamwork behavior assessment based on rater expertise.

Main Methods:

• Data collected during 12 elective surgical procedures.
• Rater pairs consisted of expert/expert (N=6) and expert/novice (N=6).
• Five teamwork behaviors (communication, coordination, leadership, monitoring, cooperation) were observed and scored pre, intra, and postoperatively.

Main Results:

• Expert/expert pairs showed significant correlations in 12/15 behaviors, while expert/novice pairs showed correlations in only 3/15 behaviors.
• Expert/expert pairs had significant mean score differences in 3/15 behaviors, whereas expert/novice pairs showed differences in 11/15 behaviors.
• Total OTAS scores demonstrated strong correlations and no significant rating differences for expert/expert pairs, contrasting with expert/novice pairs.

Conclusions:

• The OTAS tool demonstrates adequate construct validity, with expert raters achieving significantly more consistent scoring than novices.
• Expert raters using OTAS yield more reliable teamwork assessments compared to novice raters.
• Further research should investigate the learning curve for novice OTAS users and its relationship with technical skills and crisis response.