Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
Theory of Attribution II: Kelley's Covariation Theory01:29

Theory of Attribution II: Kelley's Covariation Theory

505
Attribution theory plays a crucial role in social psychology, helping to explain how individuals interpret the causes of behavior. One prominent model within this field is Harold Kelley's covariation theory, which provides a systematic approach to determining whether internal traits or external circumstances drive a person's actions. The model posits that individuals rely on three key types of information—consensus, consistency, and distinctiveness—to make these judgments.Consensus:...
505
Halo Effect01:27

Halo Effect

412
The halo effect is a cognitive bias in which an individual's overall impression influences judgments about their specific traits. This psychological phenomenon leads people to associate positive characteristics with those they perceive as generally good and negative characteristics with those they view as bad. This effect is particularly influential in social perception, professional evaluations, and decision-making processes.The Psychological Basis of the Halo EffectThe halo effect is rooted...
412
Measures of Intelligence01:29

Measures of Intelligence

8.3K
Psychologists measure intelligence by using standardized tests that produce a score known as the intelligence quotient or IQ. To understand IQ tests, it's important to recognize the key principles behind their construction: validity, reliability, and standardization.
Validity refers to how well a test measures what it claims to measure. An intelligence test should accurately assess intelligence rather than another characteristic, like anxiety. Criterion validity is one way to evaluate this;...
8.3K
Obedience01:08

Obedience

35.2K
According to obedience research, we may harm others under the forceful pressures of an authority figure (Milgram, 1974). How about if the inappropriate orders were delivered with less force? The increasing interdependence between nurses and physicians compelled Hofling and his colleagues to explore nurses’ reactions to a potentially harmful medical request made by the perceived authority figure, the doctor (Hofling, Brotzman, Dalrymple, Graves, & Pierce, 1966). In this situation,...
35.2K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Building CAR-E: A Novel Artificial Intelligence Agent for Coaching Conversations.

Perspectives on medical education·2026
Same author

Investigating a Model to Predict Milestones From Entrustable Professional Activity Levels for Medicine-Pediatrics Residents.

Academic pediatrics·2026
Same author

Getting Real(ist) With Program Evaluation.

Hospital pediatrics·2026
Same author

Interpersonal Communication and Maternal Behavioral Practice: A Case Study of Explanatory Causal Machine Learning in Nepal.

The Journal of nutrition·2026
Same author

CBME As a Philosophy of Training: Balancing Fidelity and Flexibility in Emergency Medicine.

AEM education and training·2026
Same author

Developing Resident-Sensitive Quality Measures for Internal Medicine.

JAMA network open·2026

Related Experiment Video

Updated: Jan 18, 2026

Development of a Virtual Reality Assessment of Everyday Living Skills
10:32

Development of a Virtual Reality Assessment of Everyday Living Skills

Published on: April 23, 2014

19.0K

A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments.

Matthew Kelleher1, Benjamin Kinnear, Dana Sall

  • 1M. Kelleher is assistant professor of medicine and pediatrics and associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. B. Kinnear is assistant professor of medicine and pediatrics and associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. D. Sall is assistant professor of medicine and associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio. D. Schumacher is associate professor of pediatrics, Cincinnati Children's Hospital Medical Center and University of Cincinnati College of Medicine, Cincinnati, Ohio. D.P. Schauer is associate professor of medicine and associate program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0003-3264-8154. E.J. Warm is professor of medicine and program director, Department of Internal Medicine, University of Cincinnati College of Medicine, Cincinnati, Ohio; ORCID: https://orcid.org/0000-0002-6088-2434. B. Kelcey is associate professor of quantitative research methodologies, Department of Education, University of Cincinnati, Cincinnati, Ohio.

Academic Medicine : Journal of the Association of American Medical Colleges
|October 1, 2019
PubMed
Summary

This study found that faculty ratings in residency programs have fair reliability, with raters being the main source of variance. Increasing the number of assessments and raters can significantly improve overall reliability for workplace-based assessments.

More Related Videos

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
06:02

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

Published on: October 6, 2020

2.6K
Assessing Working Memory in Children: The Comprehensive Assessment Battery for Children – Working Memory (CABC-WM)
09:05

Assessing Working Memory in Children: The Comprehensive Assessment Battery for Children – Working Memory (CABC-WM)

Published on: June 12, 2017

30.7K

Related Experiment Videos

Last Updated: Jan 18, 2026

Development of a Virtual Reality Assessment of Everyday Living Skills
10:32

Development of a Virtual Reality Assessment of Everyday Living Skills

Published on: April 23, 2014

19.0K
Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
06:02

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

Published on: October 6, 2020

2.6K
Assessing Working Memory in Children: The Comprehensive Assessment Battery for Children – Working Memory (CABC-WM)
09:05

Assessing Working Memory in Children: The Comprehensive Assessment Battery for Children – Working Memory (CABC-WM)

Published on: June 12, 2017

30.7K

Area of Science:

  • Medical Education
  • Assessment in Healthcare

Background:

  • Workplace-based assessment systems are crucial for resident training.
  • Entrustment scales are increasingly used to evaluate resident competency.
  • Generalizability theory (G-theory) is a statistical framework for analyzing reliability in such systems.

Purpose of the Study:

  • To evaluate the reliability of an entrustment-derived workplace-based assessment system.
  • To identify sources of variance within the assessment system.
  • To determine the impact of different assessment parameters on reliability.

Main Methods:

  • Utilized generalizability theory (G-theory) and decision study.
  • Analyzed 166,686 observable practice activity (OPA) ratings from July 2012 to December 2016.
  • Conducted time-specific and longitudinal G-theory analyses on residency program data.

Main Results:

  • Raters constituted the largest source of variance (37% time-specific, 23% longitudinal).
  • Resident performance was the second largest variance source (19% time-specific).
  • Reliability was ~0.40 monthly and ~0.63 over 36 months; could reach 0.76 with more raters/assessments.

Conclusions:

  • The 36-month assessment program demonstrated fair reliability.
  • Increasing the number of faculty raters and assessments per month is essential.
  • Multiple observations by multiple raters are needed to enhance assessment reliability.