What is the impact of commercial test preparation courses on medical examination performance?

Commercial test preparation courses are part of the fabric of U.S. medical education. They are also big business with 2,000 sales for 1 firm listed at nearly $250 million. This article systematically reviews and evaluates research published in peer-reviewed journals and in the "grey literature" that addresses the impact of commercial test preparation courses on standardized, undergraduate medical examinations.

Thirteen computerized English language databases were searched using 29 search terms and search concepts from their onset to October 1, 2002. Also manually searched was medical education conference proceedings and publications after the end date; and medical education journal editors were contacted about articles accepted for publication, but not yet in print, that were deemed pertinent to this review. Studies that met three criteria were selected: (a) a commercial test preparation course or service was an educational intervention, (b) the outcome variable was one of several standardized medical examinations, and (c) results are published in a peer-reviewed journal or another outlet that insures scholarly scrutiny. The criteria were applied and data extracted by consensus of 2 reviewers. The search identified 11 empirical studies, of which 10 (8 journal articles, 2 unpublished reports) are included in this review. Qualitative data synthesis and tabular presentation of research methods and outcomes are used.

The articles and unpublished reports reveal that current research lacks control and rigor; the incremental validity of the commercial courses on medical examination performance, if any, is extremely small; and evidence in support of the courses is weak or nonexistent; almost no details are given about the form and conduct of the commercial test preparation courses; studies are confined to courses in preparation for the Medical College Admission Test, the former National Board of Medical Examiners Part 1, and the United States Medical Licensing Examination Step 1, not tests of clinical science; and that cost-benefit analyses of the test preparation courses have not been done. It is concluded that the utility and value of commercial test preparation courses in medicine have not been demonstrated, and that evaluation apprehension in the medical profession and aggressive marketing practices are most likely responsible for commercial course prosperity.