Setting standards for teaching evaluation data: an application of the contrasting groups method

Teaching evaluations are widely used in retention and remediation decisions. Typically evaluations are reviewed in a global manner and some gestalt is reached.

Apply the Contrasting Groups standard setting methodology to faculty teaching dossiers, to examine resulting "pass-rates" and precision of the decisions.

Ten faculty judges set standards for teaching dossiers using a Contrasting Groups approach. Blinded dossiers summarizing clinical (N = 47) and classroom teaching (N = 37) were sorted into piles labeled Unsatisfactory, Satisfactory, Excellent, and Superior. Cut-points were the midpoints between aggregated judge-level mean performances of dossiers placed within adjacent levels.

For the total faculty, the percentage assigned to groups labeled Unsatisfactory through Superior, were 4.1%, 5.9%, 26.1%, and 63.9% and 6.6%, 24.8%, 44.1%, and 24.5% for clinical and classroom teaching respectively. Standard error of measurement was between .20 and .25 with 5 to 7 judges.

Standard setting methods applied to faculty evaluation data produce precise results. Future work can examine standard stability and acceptability, and methods to combine across teaching venues.