Cohen's Method and Its Variations in Standard Setting for Medical Examinations: An Australian Multi-Institutional...

Area of Science:

• Medical Education
• Assessment and Evaluation
• Higher Education

Background:

• Standard setting is crucial for evaluating medical student competency for clinical practice.
• No single standard-setting method is universally adopted, necessitating research into current practices.
• This study addresses the need to understand standard-setting methods in Australian medical schools.

Purpose of the Study:

• To identify standard-setting methods used in Australian medical schools.
• To analyze the benefits, limitations, and implementation challenges of these methods.
• To inform a proposal for standard-setting method changes at James Cook University.

Main Methods:

• A survey was distributed to assessment leads at all 21 Australian medical schools.
• The survey gathered data on current standard-setting methods, challenges, and satisfaction.
• Data were analyzed using descriptive statistics and thematic analysis.

Main Results:

• All 21 schools participated, reporting diverse standard-setting methods.
• Cohen/Modified Cohen was the most prevalent method (n=10), with high satisfaction (mean 83%).
• Challenges included panel recruitment, resource management, and assessment quality maintenance, highlighting the need for stakeholder involvement and resources.

Conclusions:

• The Cohen method is the most frequently used and preferred standard-setting approach for exit-level written exams in Australian medical schools.
• Its efficiency and adaptability contribute to its popularity.
• Adopting the Cohen/Modified Cohen method could align James Cook University's practices with national trends.