Test performance of a local examination at a nonlocal site for evaluation of surgical clerks

Surgical clerkships frequently use locally prepared examinations or nationally available test item banks as an alternative to Surgery Shelf Examinations from the National Board of Medical Examiners (SSNBME) for student evaluation. This study examines performance of a well-designed, item-analyzed local examination (available nationally through the Association for Surgical Education) at a nonlocal site. A 100-item test with a stratified sample from a 442-item bank was administered to 72 third-year students in addition to the SSNBME. The Pearson product moment correlation coefficient between the two examinations was 0.61 (p less than 0.001). Test performance between the local site and nonlocal site was compared by the Kuder-Richardson formula 20 for subtest reliability (internal consistency) and difficulty index (percent correct responses) and the point biserial correlation coefficient (correlation of a student's performance on one item with performance on the rest of the items) for test item analysis. Data show that despite fair correlation with SSNBME, significant deteriorations in both test item performance and reliability occur with the use of a local examination at a nonlocal site. This likely results from problems with content validity at the nonlocal site. Clerkships that use local examinations or national test bank items are strongly advised to evaluate test performance and revise appropriately before using results for formal student evaluation.