Related Experiment Video
Updated: May 29, 2026
08:33
A Cross-Disciplinary and Multi-Modal Experimental Design for Studying Near-Real-Time Authentic Examination Experiences
Published on: September 4, 2019
Evaluating construct equivalence and criterion-related validity for repeat examinees on a standardized patient
Mark R Raymond1, Nilufer Kahraman, Kimberly A Swygert
1National Board of Medical Examiners, Philadelphia, Pennsylvania 19104, USA. mraymond@nbme.org
Summary
Scores for repeat examinees on standardized patient (SP) assessments are valid on their second attempt. Repeat examinees
Area of Science:
- Medical Education
- Standardized Patient Assessments
- Psychometrics
Background:
- Previous studies show significant score improvements for examinees repeating standardized patient (SP) assessments.
- Research suggests these gains are not due to case memorization.
- The empirical validity of scores for repeat examinees has not been previously investigated.
Purpose of the Study:
- To compare the internal (construct) and external (criterion-related) validity of scores for single-take versus repeat examinees on the United States Medical Licensing Examination Step 2 Clinical Skills (CS) exam.
- To investigate the factor structure and correlations with written medical knowledge assessments for both groups.
Main Methods:
- Analysis of test scores from 12,090 single-take and 4,030 repeat examinees of the USMLE Step 2 CS exam (2007-2009).
- Confirmatory factor analysis of four Step 2 CS performance domains.
- Evaluation of correlations between Step 2 CS scores and three written medical knowledge assessments (USMLE Steps 1, 2 CK, and 3).
Main Results:
- The factor structure for repeat examinees on their first attempt differed significantly from single-take examinees.
- The factor structure for repeat examinees became more similar to single-take examinees by their second attempt.
- Scores on the second attempt by repeat examinees showed higher correlations with all three external written assessments.
Conclusions:
- The findings support the validity of scores for repeat examinees on their second attempt.
- The study provides empirical evidence for the reliability and accuracy of SP assessment scores for repeat test-takers.
Related Concept Videos
01:29Reliability and Validity
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
01:15Bioequivalence Experimental Study Designs: Repeated Measures, Cross-Over, Carry-Over, and Latin Square Designs
Bioequivalence experimental study designs play a pivotal role in testing the effectiveness of various treatments. Key among these are the repeated measures, cross-over, carry-over, and Latin square designs. In the repeated measures design, each subject receives all treatments, allowing for temporal comparisons. This type of design is useful in reducing variability but requires careful planning to avoid bias.The cross-over design, an economical method, involves sequential administration of...
01:09Bioequivalence of Drugs: Drugs with Multiple Indications
The concept of therapeutic equivalence (TE) in drugs with multiple indications is complex. A generic drug may be therapeutically equivalent to a brand-name product for one specific indication, but this doesn't necessarily mean it's equivalent for all other indications. Evidence of TE in one patient group and bioequivalence shown in healthy volunteers can support—but not confirm—TE for other indications. However, definitive proof requires individual clinical studies for each indication due to...
01:05Data Collection III
The physical assessment examines the patient for objective data that defines the patient's condition, and aids in formulating the nursing care plan. The purpose of physical assessment is a health status appraisal, which includes identifying health problems, and establishing a database for nursing intervention.
The principles to begin the physical assessment include conducting a comprehensive or problem-related history in a quiet, well-lit room, emphasizing privacy and comfort for the patient.
The principles to begin the physical assessment include conducting a comprehensive or problem-related history in a quiet, well-lit room, emphasizing privacy and comfort for the patient.
01:16Crossover Experiments
Crossover experiments, also called the repeated-measurements design, is a study design in which all experimental units are exposed to all treatments in different periods. Crossover experiments are generally used in psychology, the pharmaceutical industry, agriculture, and medicine.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
01:22Self-Report Tests of Personality
Self-report inventories are objective personality assessments that use multiple-choice items or numbered scales, typically ranging from 1 (strongly disagree) to 5 (strongly agree). They are often called Likert scales after Rensis Likert. These inventories are widely used due to their ease of administration and cost-effectiveness. One of the most prominent examples is the Minnesota Multiphasic Personality Inventory (MMPI), initially developed in the 1940s to assess abnormal personality traits.