Validity evidence for simulator-based obstetric ultrasound competency assessment tool: a multi-center study

Area of Science:

• Medical Simulation
• Ultrasound Education
• Competency Assessment

Background:

• Objective assessment of obstetric ultrasound skills is crucial for patient safety and training effectiveness.
• Existing assessment methods may lack standardized validation or rely on subjective evaluations.
• Simulator-based training offers a controlled environment for skill acquisition and assessment.

Purpose of the Study:

• To gather comprehensive validity evidence for the simulator-based Obstetric Ultrasound Competency Assessment Tool (OUCAT).
• To establish the reliability and accuracy of OUCAT in evaluating sonographers' obstetric ultrasound proficiency.
• To provide a standardized tool for assessing competency in obstetric ultrasound procedures.

Main Methods:

• Collected validity evidence for OUCAT following the Standards for Educational and Psychological Testing.
• Assessed 89 sonographers (novices, experienced trainees, experts) across three centers.
• Evaluated content validity, response process, internal structure (reliability), relations to other variables, and consequences (pass/fail level).

Main Results:

• OUCAT comprised 123 items, with 117 effectively distinguishing novices from experts (P<0.05).
• High internal consistency (Cronbach's α = 0.978), inter-rater reliability (0.868-0.937), and test-retest reliability (0.732) were observed.
• Experts significantly outperformed experienced trainees, who in turn outperformed novices (P<0.001); pass rate for novices was 0%, experts 100%.

Conclusions:

• Simulator-based OUCAT demonstrates robust reliability and validity for assessing obstetric ultrasound skills.
• The tool effectively differentiates skill levels among sonographers, supporting its use in competency evaluation.
• OUCAT provides a valuable, validated instrument for standardizing and improving obstetric ultrasound training and assessment.