Effects of a long term faculty development program on improvement in quality of MCQs: an impact evaluation study

The faculty development programs associated with positive outcomes are the ones that are designed in accordance with organizational needs, are based on a theoretical framework, target inter-professional learning and use multiple teaching strategies to teach a single skill. Rather than being short, one-time events, these programs are long term with contextual engagement of participants. To meet this challenge, a six-month long certificate course in health professions education, based on constructivist theory, was designed. The objective of this study was to evaluate the impact of this focused training, at Kirkpatrick level II, on learning of item writing skills by the participants.

This quasi-experimental study was conducted from January 2019 to June 2020 at National University of Medical Sciences, Pakistan. A total of 133 faculty members were enrolled in the program. Of these, data from 124 students (75 male and 49 female participants), who had passed our CHPE program, was included in the study. The longitudinal engagement comprised of three steps. In step 1(pre-intervention) participants submitted 5MCQs each, resulting in a total of 620 MCQs. For the step 2 (intervention), a six-hour workshop on writing single best MCQs with peer and faculty feedback during the five-day long face to face session was conducted for all the participants. Subsequently four courses including learning theories, curriculum planning, teaching and learning and assessment were delivered over the course of next six months with special focus on building participants' capacity on blueprinting so as to highlight the linkages between various courses and their significance through a single snapshot document. An assignment on writing single best MCQs was given as part of course on assessment and faculty gave individual feedback to participants. Finally, for the step 3- post intervention, the skills were tested in the end of program exam. A validated checklist was used to score the items on quality parameters. A total of 1800 MCQ items (600 at each step) were analyzed at these three points of intervention; and the scores obtained were compared to assess if the item writing skills had improved.

The average scores across three steps showed an increasing trend. The Friedman ANOVA test results indicated that there was a statistically significant difference across the three time points with a value of test statistics χ² (2, n = 600 = 955.86, P < 0.05). The Wilcoxon signed rank test showed significant differences between each pair of steps, supporting the finding of an improvement trend in the scores across all steps. Step 3 had the highest value of mean ranks (20.24 ± 0.05) and a higher median score (Md = 21). Bonferroni correction showed that at the threshold level of 0.0167, all three comparisons are still statistically significant.

Contextual engagement of participants through a longitudinal faculty development program, incorporating varied teaching techniques like individual and group work, practice and prompt feedback and peer review, improves participants' MCQ writing skills, such that they construct MCQs that assess higher cognitive skills with fewer item writing flaws.