A validity study of COMLEX-USA Level 3 with the new test design

Area of Science:

• Medical Education Assessment
• Psychometrics
• Osteopathic Medical Licensure

Background:

• The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) for osteopathic physician licensure.
• The COMLEX-USA Level 3 (L3) examination was redesigned in September 2018 to a two-day computer-based format, incorporating multiple-choice questions (MCQ) and clinical decision-making (CDM) components.
• Continued validation of the L3 examination under its new design is crucial for accurate score interpretation and utilization.

Purpose of the Study:

• To gather validity evidence for the COMLEX-USA Level 3 (L3) examination scores under the revised two-day computer-based format.
• To apply Kane's validity framework, examining evidence from scoring, generalization, extrapolation, and implication/decision components.
• To ensure the L3 examination accurately measures the intended constructs for osteopathic medical licensure.

Main Methods:

• Utilized Kane's validity framework, collecting data from 8,366 first-time COMLEX-USA Level 3 (L3) examinees (September 2018 - December 2019).
• Assessed content coverage, scoring procedures, reliability, decision accuracy, score associations (MCQ/CDM components, L3/L2-PE), and subgroup performance.
• Employed automated test assembly (ATA) for form parallelism and a criterion-referenced standard-setting process for pass/fail determination.

Main Results:

• Assessment forms demonstrated parallelism via automated test assembly (ATA); scoring adhered to quality standards.
• High inter-rater reliability for short answer (SA) ratings and high decision accuracy/consistency for pass/fail classifications were observed.
• Statistically significant positive associations were found between MCQ and CDM components, and between L3 scores and COMLEX-USA Level 2 Performance Evaluation (L2-PE) domain scores, aligning with measurement expectations. Subgroup analyses revealed expected minor score variations.

Conclusions:

• The study provides substantial validity evidence for the redesigned COMLEX-USA Level 3 (L3) examination, supporting its use for osteopathic medical licensure.
• Ongoing evaluation and evidence collection are essential for maintaining the validity argument of the COMLEX-USA examination series.
• The findings affirm the defensibility of the criterion-referenced standard-setting procedure used for L3 pass/fail decisions.