Improved Activity Evaluations: An Iterative Process Using the Rasch Model

Traditional evaluation models, often linear and outcome-focused, are increasingly inadequate for the complexities of modern medical education, which demands more comprehensive and nuanced assessment approaches.

A standardized continuing professional development activity evaluation instrument was developed and implemented. An iterative process was performed, using a repeat Rasch analysis, to improve reliability of the evaluation instrument. Category Probability Curves and Test Information Function were generated by the Rasch analysis to refine the construction of the assessment. All educational activities completed between 2022 and 2024 were eligible for inclusion. The study incorporated a diverse range of educational activities and included multiple health care professions.

The pilot analysis included 250 educational activities with 26,554 individual learners completing evaluations for analysis. Initial Rasch findings demonstrated a need to remove redundancies and change from a five to four-point rating scale. The final instrument validation included 21 activities and 529 learners. Improvement was seen in reliability after modifications, with an increase in Cronbach alpha from 0.72 to 0.80.

Use of psychometrics to improve assessments can yield a more reliable and less redundant evaluation instrument. This research demonstrates a psychometrically informed, flexible evaluation tool that can inform future educational efforts and serve as a data driven metric to enhance the quality of continuing professional development programs.