Using standardized patients in licensing/certification examinations: comparison of two tests in Canada

A new licensing examination for family physicians was introduced in the Province of Quebec, Canada, in 1990. It contains a newly developed, standardized, patient-based objective structured clinical examination (OSCE) used in complementarity with two other instruments (Short Answers Management Problems [SAMPs] and Simulated Office Orals [SOOs]) that have been used for many years by the College of Family Physicians of Canada (CFPC) for certification purposes. This research studied the criterion validity of the OSCE using the last instruments as criteria.

The results of the 172 spring 1994 candidates were analyzed. Cronbach alpha reliability coefficients were computed. Pearson's correlation coefficients and regression analysis measured the extent to which scores on the new Quebec OSCE correlated with the scores on the CFPC instruments.

Correlation coefficients were significant between the Quebec OSCE scores and the CFPC examination total scores (.700), the SAMPs scores (.634), and the SOOs scores (.514). Multiple regression analysis confirmed that scores on the Quebec OSCE explained a significant but still limited proportion of the variation in scores of the CFPC examination.

This study supports the criterion validity of the Quebec OSCE used in the licensing examination. However, differences in the skills assessed by each instrument and in the clinical situations presented exist between the Quebec instrument and the two national instruments. These differences, plus the variation in the scores obtained by the candidates on the three instruments, support the decision of the Quebec licensing organization to use them in complementarity.