Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.

Area of Science:

• Medical Education
• Surgical Training
• Assessment Methodology

Background:

• Proficient performance in surgical tasks relies on well-developed mental representations.
• Video Commentary (VC) is a novel assessment tool designed to evaluate these mental representations in surgical tasks.
• Understanding how residents approach VC is crucial for its validation and effective use.

Purpose of the Study:

• To investigate the response processes of surgical residents during Video Commentary (VC) assessments.
• To determine if residents' understanding and application of VC align with its intended purpose of measuring mental representations.
• To identify factors influencing residents' performance and response strategies in VC.

Main Methods:

• A qualitative study employing grounded theory was conducted with surgical residents.
• Semi-structured interviews explored residents' understanding of and approach to VC.
• Likert-type questions assessed residents' perceived purpose of VC.
• Analysis of narrative comments and quantitative data informed a theoretical model of response processes.

Main Results:

• Residents generally perceive VC's purpose as intended by educators.
• Response processes are influenced by test characteristics, resident understanding, and personal factors.
• Four primary response strategies were identified: focus on speed, points, logic, and relevance.
• Residents believe VC scores reflect quick speaking, quick thinking, and anatomy knowledge.
• Anterior residents focus on facts, while senior residents provide comprehensive descriptions.

Conclusions:

• Surgical residents generally grasp the educational intent of VC.
• Divergent response strategies (speed, points, logic, relevance) may pose a threat to assessment validity.
• These identified response strategies offer insights applicable to other clinical skills assessments.