Related Experiment Video
Updated: Oct 16, 2025
08:33
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
7.2K
Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for
Nina W Zhao1, Lindsey M Haddock2, Bridget C O'Brien2
1Department of Otolaryngology-Head and Neck Surgery, University of California - San Francisco, San Francisco, California; Department of Otolaryngology-Head and Neck Surgery, University of California - Davis, Sacramento, California.
Journal of Surgical Education
|October 20, 2021
Summary
This study explored how surgical residents and faculty interpret workplace-based assessments (WBAs). Findings reveal that while interpretations are similar, cognitive, contextual, and emotional factors influence ratings, impacting formative feedback validity.
Area of Science:
- Medical Education Research
- Surgical Training Assessment
- Qualitative Research Methods
Background:
- Workplace-based assessments (WBAs) are crucial for formative feedback in surgical training.
- Limited evidence exists on how users interpret WBAs and make rating decisions (response processes).
- Understanding response processes is vital for validating the use of WBAs in surgery.
Purpose of the Study:
- To explore validity evidence based on response processes for a surgical WBA.
- To investigate how faculty and residents interpret assessment questions and rating categories.
- To identify factors influencing rating decisions in surgical WBAs.
Main Methods:
- Semi-structured interviews were conducted with surgical residents and faculty using the System for Improving and Measuring Procedural Learning (SIMPL).
- Interviews focused on the interpretation of assessment questions (autonomy, performance, complexity) and answer choices.
- Directed qualitative content analysis was used to analyze interview transcripts and identify themes.
Main Results:
- Users demonstrated similar content-level interpretations of SIMPL questions and answer choices.
- Rating decisions were influenced by diverse cognitive, behavioral, and emotional processes.
- Contextual factors and interpretive tensions significantly impacted user ratings and introduced uncertainty.
Conclusions:
- Response processes are essential for validating the formative use of WBAs in surgical education.
- Assessment validity requires analysis beyond content, considering contextual influences and interpretive challenges.
- Further research is needed to ensure WBAs effectively meet formative assessment needs in surgery.
Keywords:
Interpersonal and Communication SkillsPractice-Based Learning and ImprovementSystems-Based Practiceassessmentfeedbackresponse processvalidityworkplace-based assessmentMore Related Videos
Related Concept Videos
01:03Data Validation
5.9K
Data validation is an essential part of a comprehensive assessment. Validation is confirming or verifying and opening the door to gathering more assessment data as it clarifies vague or unclear data. The process of checking and verifying the collected information is called data validation. The primary purpose of data validation is to ensure data is as free from error, bias, and misinterpretation as possible.
Nursing assessment guides are generally based on holistic models rather than medical...
Nursing assessment guides are generally based on holistic models rather than medical...
5.9K
01:29Reliability and Validity
13.3K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.3K
01:29Self-Evaluation Maintenance Model
42
The Self-Evaluation Maintenance (SEM) model offers a psychological framework to understand how individuals’ self-esteem is influenced by the achievements of others, particularly those with whom they share close personal bonds. The SEM model operates when personal rather than social identity guides individuals. Central to this model is the notion that individuals have an inherent desire to preserve a favorable self-image, which is continuously shaped by interpersonal comparisons and...
42
01:24Sources of Self-Esteem II: Performance Feedback
18
Self-esteem is intricately tied to our perception of competence and our ability to exert control over our lives. One of the primary sources of this perception is performance feedback — the ongoing evaluation of our actions in terms of success and failure. According to Franks and Marolla (1976), people derive self-worth from experiencing themselves as causal agents, capable of achieving goals and overcoming obstacles. This process nurtures a critical component of self-esteem:...
18
01:27Halo Effect
48
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...
48
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
2.1K
Evaluation of the teaching process enables the nurse to determine if the patient's learning needs were met and if training was effective. If the expected outcomes are not met, the care plan is revised, and additional education or reinforcement is provided. Nurses can ask questions after the session or obtain feedback to assess the patient's understanding of the topic.
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
2.1K