Assessing the dependability of faculty observations using the generalizability coefficient

Evaluating performance of dental students is often accomplished by faculty observation. Ratings are assigned and later translated into scores which form the basis for important decisions such as admission to dental school, course grades, or admission to graduate programs. Situational factors such as variability among faculty observers or day-to-day fluctuations in a student's performance may influence observational ratings. These situational factors can introduce error into the ratings and thereby reduce the dependability of decisions based upon the ratings. Generalizability coefficients are a recent development in educational measurement which can aid the dental educator in evaluating the degree of error present in ratings to be used for a specific purpose. In this study, dental school applicant ratings from an admissions interview are used to demonstrate the usefulness of generalizability coefficients in assessing the impact of interviewer disagreements on decisions based on these scores.