Assessing context effects on test validity of the National Board Dental Examination Part I

In support of actions taken by the Joint Commission on National Dental Examinations, two changes--the inclusion of testlet items and the random presentation of items in an interdisciplinary format--were made to enhance the test validity of the National Board Dental Examination Part I in 2007. As a result, the examination was changed from a conjunctive to a comprehensive format. It was assumed that validity would be enhanced with regard to the examination's internal structure, while not disturbing item performance and examinee score. This study of the results found that 1) three underlying variables were extracted from the conjunctive Part I but only two underlying variables from the comprehensive Part I and 2) the differences in item performance and examinee score were generally small in effect size across formats. Factor analyses revealed that Part I was more discipline-sensitive for the conjunctive format but more item format-sensitive for the comprehensive format. The revision of Part I changed the nature of the examination from a discipline-based format to a more clinically relevant, interdisciplinary format, a favorable outcome anticipated by the Joint Commission. The results of this study provide evidence supporting the validity of the revised Part I examination for its intended purpose in the licensure process.