Analyzing the sources of unreliability in fixed prosthodontics mock board examinations

Mock board examinations in fixed prosthodontics were submitted to generalizability analysis in order to determine which sources of unwanted variance of measurement contribute to grade decisions and whether this lack of reliability is of practical significance. Students completed approximately three fixed prosthodontics test cases during their final year of clinic, and each case was scored by two faculty members. Of the subsamples of students where two test cases (trials) per student were graded by the same two raters, the subsamples with the highest and the lowest inter-rater reliability coefficients were chosen. Typical generalizability coefficients (reliability considering both raters and trials as sources of error) are much lower than the inter-rater reliability estimate, and the standard error of measurement is 80 percent of a grade interval on a five-point scale. In all analyses, the largest source of variance was the student-by-trial interaction, accounting for about 80 percent of the standard error of measurement or one-half a grade on a five-point scale. Even in the subsample with lowest inter-rater reliability, rater, rater-by-student interaction, and rater-by-trial interaction made no contribution to measurement error. Nor did students show evidence of improving over time. There is no possible improvement through the use of rater calibration or additional raters that would equal the improvement made by using two test cases rather than one. The concept of gradient of generalizability is introduced, and implications for initial licensure examinations are discussed.