A Decision-oriented Approach to Evaluating a Leadership Curriculum in Fellowship

Graduate Medical Education (GME) emphasizes the role of the annual program evaluation to identify opportunities, create action plans, and track improvements longitudinally. There is a lack of a systematic approach to the evaluation of educational curricula. Comprehensive curriculum evaluation can inform the educators about specific modifications to achieve high standards, desired outcomes, and the anticipated objectives.

To evaluate a leadership in quality improvement program in a pulmonary/critical care fellowship training program using the context, input, process, product (CIPP) model. The CIPP model, given its focus on evaluating different aspects of a program, provides concrete and targeted feedback to guide improvement decisions.

Evaluation questions addressing the four focused areas were created, pilot tested, and revised. The questions were framed toward optimization of alignment (e.g., program activities with stated objectives, program goals with theoretical perspective, program curriculum with trainee needs) and gaining information about the efficacy of the program in achieving the desired outcomes. To enhance the validity of the results, we triangulated the data-gathering approach by administering surveys and conducting interviews and focus groups by random selection from the eligible participants. Qualitative data were transcribed, coded, and categorized into themes aligning with the four aspects of the CIPP model.

We interviewed 9 participants and conducted three focus groups with 20 participants. The surveys provided vital quantitative information that was cross-verified with the qualitative data; 23 of the 25 (92%) participants completed the survey. The results of qualitative thematic analysis were organized in the CIPP format. The context evaluation of the program revealed that the fellows and faculty were unfamiliar with the guiding principles of the course. The input evaluation highlighted the competing interests that hampered the engagement of the fellows during the evening weekly report-outs. The process evaluation revealed clustering of didactic sessions at the start of the course. The product evaluation stressed the difficulty in completing the quality improvement projects in the allotted timeframe.

Conducting a robust evaluation of an educational curriculum provides insights into gaps in the various stages of the program. Time and resources needed for conducting evaluation by using the CIPP model should be considered.