Setting standards for teaching evaluation data: an application of the contrasting groups method.

Area of Science:

• Medical Education
• Faculty Development
• Higher Education Assessment

Background:

• Faculty teaching evaluations are crucial for retention and remediation decisions.
• Current evaluation methods often rely on subjective, global assessments.
• There is a need for more objective and precise methods in faculty performance review.

Purpose of the Study:

• To apply the Contrasting Groups standard-setting methodology to faculty teaching dossiers.
• To determine the resulting pass-rates and decision precision using this method.
• To evaluate the effectiveness of a structured approach for faculty performance assessment.

Main Methods:

• Ten faculty judges utilized the Contrasting Groups approach to set standards for teaching dossiers.
• Blinded dossiers, encompassing clinical (N=47) and classroom (N=37) teaching, were categorized.
• Categories included Unsatisfactory, Satisfactory, Excellent, and Superior, with cut-points based on aggregated judge performance.

Main Results:

• For clinical teaching, distribution was 4.1% Unsatisfactory, 5.9% Satisfactory, 26.1% Excellent, and 63.9% Superior.
• For classroom teaching, distribution was 6.6% Unsatisfactory, 24.8% Satisfactory, 44.1% Excellent, and 24.5% Superior.
• The standard error of measurement ranged from 0.20 to 0.25 with 5 to 7 judges, indicating high precision.

Conclusions:

• Standard-setting methodologies, like Contrasting Groups, yield precise outcomes when applied to faculty evaluation data.
• Future research should investigate the stability and acceptability of these standards.
• Developing methods to integrate evaluations across different teaching venues is recommended.