Related Experiment Video
Updated: May 8, 2026

Evaluating Tests of Cognition using a Computerized Touch-Sensitive Tablet, Eye Tracking, and Functional Magnetic Resonance Imaging
Published on: January 30, 2026
Comparing diagnostic performance and the utility of clinical vignette-based assessment under testing conditions
Jonathan S Ilgen1, Judith L Bowen, Lucas A McIntyre
1Dr. Ilgen is assistant professor, Division of Emergency Medicine, University of Washington, School of Medicine, Seattle, Washington. Dr. Bowen is professor, Department of Medicine, Oregon Health & Science University, School of Medicine, Portland, Oregon. Mr. McIntyre is a student, University of Washington, School of Medicine, Seattle, Washington. Dr. Banh is assistant clinical professor, Department of Emergency Medicine, University of California, San Francisco, School of Medicine, Fresno, California. Dr. Barnes is assistant professor, Department of Emergency Medicine, University of California, Davis, School of Medicine, Sacramento, California. Dr. Coates is professor of clinical medicine, Department of Emergency Medicine, University of California, Los Angeles, David Geffen School of Medicine, and Los Angeles Biomedical Research Institute, Los Angeles, California. Dr. Druck is associate professor, Department of Emergency Medicine, University of Colorado, School of Medicine, Denver, Colorado. Dr. Fix is assistant professor, Division of Emergency Medicine, University of Utah, School of Medicine, Salt Lake City, Utah. Dr. Rimple is associate professor, Department of Emergency Medicine, University of New Mexico, School of Medicine, Albuquerque, New Mexico. Dr. Yarris is associate professor, Department of Emergency Medicine, Oregon Health & Science University, School of Medicine, Portland, Oregon. Dr. Eva is professor and director of education research and scholarship, Department of Medicine, and senior scientist, Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada.
Purpose:
Although decades of research have yielded considerable insight into physicians' clinical reasoning processes, assessing these processes remains challenging; thus, the authors sought to compare diagnostic performance and the utility of clinical vignette-based assessment under testing conditions designed to encourage either automatic or analytic thought.
Method:
This 2011-2012 multicenter randomized study of 393 clinicians (medical students, postgraduate trainees, and faculty) measured diagnostic accuracy on clinical vignettes under two conditions: one encouraged participants to give their first impression (FI), and the other led participants through a directed search (DS) for the correct diagnosis. The authors compared accuracy, feasibility, reliability, and relation to United States Medical Licensing Exam (USMLE) scores under each condition.
Results:
A 2 (instructional condition) × 2 (vignette complexity) × 3 (experience level) analysis of variance revealed no difference in accuracy as a function of instructional condition (F[1,379] = 2.44, P = .12), but demonstrated the expected main effects of vignette complexity (F[1,379] = 965.2, P < .001) and experience (F[2,379] = 39.6, P < .001). Pearson correlations revealed greater associations between assessment scores and USMLE performance in the FI condition than in the DS condition (P < .001). Spearman-Brown calculations consistently indicated that alpha ≥ 0.75 could be achieved more efficiently under the FI condition relative to the DS condition.
Conclusions:
Instructions to trust one's first impres-sions result in similar performance when compared with instructions to consider clinical information in a systematic fashion, but have greater utility when used for the purposes of assessment.
Related Concept Videos
Critical Thinking II
Patient-centered Care
Critical Thinking I
Automated Microbial Diagnostics
Decision Making: Traditional Method
First, a specific claim about the population parameter is decided based on the research question and is stated in a simple form. Further, an opposing statement to this claim is also stated. These statements can act as null and alternative hypotheses, out of which a null hypothesis would be a...
Sensitivity, Specificity, and Predicted Value
Sensitivity is the...