Related Experiment Video
Updated: Jan 21, 2026
Project-Based Learning Guidelines for Health Sciences Students: An Analysis with Data Mining and Qualitative Techniques
Published on: December 9, 2022
Examinee Cohort Size and Item Analysis Guidelines for Health Professions Education Programs: A Monte Carlo Simulation
André-Sébastien Aubin1, Meredith Young, Kevin Eva
1A.-S. Aubin is professor, Département d'éducation et de pédagogie, Université du Québec à Montréal, Montréal, Québec, Canada. At the time of the study, he was a postdoctoral fellow, Chaire de recherche en pédagogie médicale Paul Grand'Maison de la Société des médecins de l'Université de Sherbrooke, Sherbrooke, Québec, Canada. M. Young is associate professor, Department of Medicine, and research scientist, Centre for Medical Education, McGill University, Montreal, Québec, Canada; ORCID: http://orcid.org/0000-0002-2036-2119. K. Eva is senior scientist, Centre for Health Education Scholarship, and professor, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: https://orcid.org/0000-0002-8672-2500. C. St-Onge is professor, Department of Medicine, Faculty of Medicine and Health Sciences, and Chaire de recherche en pédagogie médicale Paul Grand'Maison de la Société des Médecins, Université de Sherbrooke, Sherbrooke, Québec, Canada; ORCID: https://orcid.org/0000-0001-5313-0456.
Purpose:
Using item analyses is an important quality-monitoring strategy for written exams. Authors urge caution as statistics may be unstable with small cohorts, making application of guidelines potentially detrimental. Given the small cohorts common in health professions education, this study's aim was to determine the impact of cohort size on outcomes arising from the application of item analysis guidelines.
Method:
The authors performed a Monte Carlo simulation study in fall 2015 to examine the impact of applying 2 commonly used item analysis guidelines on the proportion of items removed and overall exam reliability as a function of cohort size. Three variables were manipulated: Cohort size (6 levels), exam length (6 levels), and exam difficulty (3 levels). Study parameters were decided based on data provided by several Canadian medical schools.
Results:
The analyses showed an increase in proportion of items removed with decreases in exam difficulty and decreases in cohort size. There was no effect of exam length on this outcome. Exam length had a greater impact on exam reliability than did cohort size after applying item analysis guidelines. That is, exam reliability decreased more with shorter exams than with smaller cohorts.
Conclusions:
Although program directors and assessment creators have little control over their cohort sizes, they can control the length of their exams. Creating longer exams makes it possible to remove items without as much negative impact on the exam's reliability relative to shorter exams, thereby reducing the negative impact of small cohorts when applying item removal guidelines.
More Related Videos
Related Concept Videos
01:26Guidelines for Nursing Documentation II
Timely documentation is crucial to ensure continuity of care for patients. Any delays in recording or reporting medical information can result in medical errors and even adverse patient outcomes. From medication administration to diagnostic test results, every detail must be accurately and promptly documented to provide the best possible care for patients.
01:06Legal Guidelines for Documentation
01:23Guidelines for Sketching a Curve
01:11Guidelines for Writing Outcome
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...
01:30Guidelines for Nursing Documentation I
Factual:
The following points emphasize the significance of upholding accurate and unbiased documentation in healthcare.
01:19Guidelines For Measuring Vital Signs
Before taking a patient's vital signs, a nurse would consider and assess the patient's comfort level and ensure appropriate equipment is available.