Understanding COMLEX-USA Level-1 as a Pass/Fail examination: impact and opportunities

In late 2020, the National Board of Osteopathic Medical Examiners (NBOME) announced that the Comprehensive Osteopathic Medical Licensing Examination (COMLEX-USA) Level 1 would discontinue reporting numeric scores, only providing a Pass/Fail designation. The National Board of Medical Examiners (NBME) changed the core reports for the United States Medical Licensing Examination (USMLE) Step-1 to Pass/Fail in early 2020. Subsequently, residency program directors were reported to migrate toward increased reliance on COMLEX-USA Level 2CE and USMLE Step 2CK scores, the prestige of medical schools, and the status of the authors of letters of reference during their review of residency applicants.

Funded by an American Association of Colleges of Osteopathic Medicine (AACOM) research grant, A.T. Still University - Kirksville College of Osteopathic Medicine (ATSU-KCOM) and the Arkansas College of Osteopathic Medicine (ARCOM) generated a series of five hypotheses to investigate the perceived impact of the transition to Pass/Fail scoring.

Both institutional review boards (IRBs) provided exempt status for this study. The hypotheses were examined through surveys distributed to current osteopathic medical students (OMS), recent graduates (classes of 2021 and 2022), academic advisors, and college of osteopathic medicine (COM) curriculum committee members in this cohort study. The investigative process consisted of two surveys: an initial survey to collect data on all five hypotheses and a focused, follow-up survey designed to better understand the impact of the scoring change. The surveys collected respondent feedback based on a Likert Scale. Standard descriptive statistics were assembled and analyzed. Qualitative responses were coded into thematic elements to examine response patterns.

The initial survey collected responses from 302 respondents (23.2 % response rate) based on a scale from 1 (strongly disagree) to 4 (strongly agree). The survey results revealed the means for the following items based on a 4-point Likert scale (1=Strongly Disagree; 2=Disagree; 3=Agree; 4=Strongly Agree): (1) Increase chances of matching into residency: 2.35; (2) Pass/Fail is a positive change: 2.71; (3) Pass/Fail will reduce competition in competitive residency programs: 1.94; (4) clinical skills will be more strongly considered: 2.60; and (5) new score reports provide a robust analysis of strengths and weaknesses in basic science: 2.24. Almost half (46 %) of the respondents expressed negative sentiments toward the Pass/Fail change. Reasons included more emphasis on COMLEX-USA Level 2CE, decreased motivation to perform on COMLEX-USA Level 1, and the perceived disadvantages that resulted during the residency match process.

Among the survey respondents, many, especially students, do not view the change as favorable. Respondents cited the primary reasons for their unfavorable perception as a lack of motivation to perform on COMLEX-USA Level 1 and the added stress of performing on COMLEX-USA Level 2CE. Additionally, respondents viewed the change as negatively impacting their audition rotations and subsequent residency competitiveness.