Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

12.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.7K
Surveys02:16

Surveys

14.8K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
14.8K
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches01:23

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

129
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
129
Measures of Intelligence01:29

Measures of Intelligence

7.3K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
7.3K
Case Studies01:22

Case Studies

11.7K
There are many research methods available to psychologists in their efforts to understand, describe, and explain behavior and the cognitive and biological processes that underlie it.
11.7K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Raising the bar: Can an Online Module With a Validated Tool Improve Clerkship Evaluations?

Journal of medical education and curricular development·2026
Same author

Does USMLE Performance Predict Residency Milestone Ratings in Medical Knowledge and Patient Care? A Multi-Site, Multi-Specialty Study.

Academic medicine : journal of the Association of American Medical Colleges·2026
Same author

Automated Alerts to Improve Timely Evaluation and Treatment of Valvular Heart Disease: The ALERT Trial.

Journal of the American College of Cardiology·2026
Same author

Does the Timing of USMLE Step 1 Affect Performance? A Comparison of Pre- and Post-Clerkship Timing in the Pass/Fail Era.

Teaching and learning in medicine·2026
Same author

Navigating challenges in competency-based graduation: accelerating medical training while ensuring rigor and readiness for residency.

Academic medicine : journal of the Association of American Medical Colleges·2026
Same author

Surgical Efficacy of Robotic Adrenalectomy: Korean Nationwide Comparative Data Analysis Focusing on Robotic Modality, Surgical Approach, and Tumor Characteristics.

Journal of laparoendoscopic & advanced surgical techniques. Part A·2026

Related Experiment Video

Updated: Jul 6, 2025

Development of a Virtual Reality Assessment of Everyday Living Skills
10:32

Development of a Virtual Reality Assessment of Everyday Living Skills

Published on: April 23, 2014

18.5K

How well do workplace-based assessments support summative entrustment decisions? A multi-institutional

Michael S Ryan1,2, Katherine A Gielissen3, Dongho Shin4

  • 1Department of Pediatrics, University of Virginia, Charlottesville, Virginia, USA.

Medical Education
|January 3, 2024
PubMed
Summary
This summary is machine-generated.

Workplace-based assessments (WBAs) for residency entry show low reliability across multiple institutions. Factors like faculty development and scale selection did not consistently improve WBA reliability for high-stakes decisions.

More Related Videos

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
08:27

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

6.9K
Using Learning Outcome Measures to assess Doctoral Nursing Education
10:07

Using Learning Outcome Measures to assess Doctoral Nursing Education

Published on: June 21, 2010

19.0K

Related Experiment Videos

Last Updated: Jul 6, 2025

Development of a Virtual Reality Assessment of Everyday Living Skills
10:32

Development of a Virtual Reality Assessment of Everyday Living Skills

Published on: April 23, 2014

18.5K
Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits
08:27

Applying an eMASS Customization Program as a Research Tool to Evaluate Consumer Benefits

Published on: September 27, 2019

6.9K
Using Learning Outcome Measures to assess Doctoral Nursing Education
10:07

Using Learning Outcome Measures to assess Doctoral Nursing Education

Published on: June 21, 2010

19.0K

Area of Science:

  • Medical Education
  • Assessment in Health Professions Education

Background:

  • Workplace-based assessments (WBAs) are crucial for evaluating Core Entrustable Professional Activities for Entering Residency (CEPAER).
  • Previous single-institution studies reported inconsistent reliability of WBAs for measuring progression towards entrustment.
  • Faculty development, rater engagement, and scale selection are potential factors influencing WBA reliability.

Purpose of the Study:

  • To conduct a multi-institutional generalisability study.
  • To determine the influence of specific factors on the reliability of WBAs.

Main Methods:

  • Analysis of 53,965 WBAs from clerkship-level students across seven institutions (2018-2020).
  • Application of generalisability theory and a cross-classified random-effects model to decompose variance components.
  • Evaluation of factors including designated assessors (DA), scale, and Entrustable Professional Activities (EPAs) on reliability, with a phi coefficient >0.7 as the threshold.

Main Results:

  • Low variance attributed to the learner (<10%) was observed in 76% of analyses, necessitating a high number of observations for reliability.
  • Reliability varied widely, with a median of 60 observations required.
  • Factors such as DA, scale, or EPA were not consistently linked to improved WBA reliability.

Conclusions:

  • WBAs across multiple sites demonstrated relatively low reliability.
  • Generalisability of WBAs may be less influenced by faculty development, rater engagement, or scale selection than previously thought.
  • While useful for formative feedback, WBAs may lack consistent reliability for high-stakes summative entrustment decisions.