Residency directors' predictions of candidates' performances on a licensing examination

To look at how well residency directors in family medicine predict the performances of their candidates on the Quebec Licensing Examination (QLEx) for family physicians.

The four family medicine program directors in Quebec were asked in the spring of 1992 to identify the ten residents from their own programs who would get the highest marks on the QLEx and the ten who would get the lowest marks. From the results of these candidates, and those of the intermediate groups defined by default, the prediction abilities of the program directors were assessed.

Descriptive statistics showed that the program directors had difficulties discriminating among the different groups and tended to collapse the predicted scores toward the total mean. Analysis-of-variance studies confirmed the absence of difference between the predicted weak and intermediate groups as well as between the predicted intermediate and strong groups for each program and for each QLEx component. Regression analysis as well as eta 2 studies showed that the program directors' prediction abilities were low for all components and represented less than 25% of the explained variance of the QLEx scores.

The residency directors did not accurately categorize their residents' performances on the QLEx. Both the evaluations of program directors and terminal examination results are complementary approaches to clinical competence assessment and should be used for licensure.