Validity evidence for summative performance evaluations in postgraduate community pharmacy education

Area of Science:

• Pharmacy education
• Competency-based assessment
• Workplace learning

Background:

• Workplace-based assessment of professional competencies is challenging.
• The Canadian Medical Education Directive for Specialists (CanMEDS) framework is a widely used model for defining specialist competencies.
• Evaluating trainee performance in postgraduate programs requires robust and valid assessment methods.

Purpose of the Study:

• To investigate the validity of summative performance evaluations (SPEs) in a postgraduate community pharmacy specialization program.
• To assess the reliability and impact of SPEs on decision-making within the program.
• To examine the construct of competence using an adapted CanMEDS framework.

Main Methods:

• A case study design incorporating quantitative and qualitative data collection.
• Confirmatory factor analysis and generalizability theory applied to SPE scores from 342 trainees over two years.
• Semi-structured interviews with 15 supervisors and the program director to explore scoring inferences and decision-making impact.

Main Results:

• The adapted CanMEDS framework demonstrated a valid seven-factor structure for competence.
• Reliability and precision of SPE measurements could not be fully determined due to study design limitations.
• Supervisor scoring standards varied, with some reluctance to fail trainees, and SPE scores had minimal influence on program director decisions.

Conclusions:

• The adapted CanMEDS framework offers a valid structure for assessing pharmacy trainee competence.
• SPEs in this program lacked established reliability and provided limited input for high-stakes decisions.
• Further investigation into potential "shadow assessment" systems within the training pharmacies is warranted.