Using chart reviews to evaluate a continuing medical education (CME) program

Context Moore's Expanded Outcomes Framework is a 7 level framework commonly used to assess the outcomes of continuing medical education (CME) programs. Levels 1 to 5 are provider-level outcomes (participation, satisfaction, knowledge, competence, and performance) while levels 6 and 7 are patient- and community-level outcomes. Chart reviews are one method to assess level 5 (performance). ECHO Ontario Chronic Pain and Opioid Stewardship ("ECHO") is a CME telementoring program that aims to increase capacity and access for primary care providers (PCPs) who manage patients with chronic pain. Objective This study uses chart reviews to evaluate ECHO's impact on PCP performance and to discuss the feasibility of performing chart reviews for evaluation purposes as per Moore's framework. Study Design Retrospective chart review Setting The practices of 12 primary care providers across Ontario who attended ECHO between June 2014 to August 2018. The inclusion criteria for PCPs was 1) attended a minimum of four ECHO sessions, and 2) clinic site must be approving of a site visit for chart reviews. Population Studied 47 patient charts were included. For each patient chart reviewed, PCPs were asked to choose patients in their practice on whom they had used ECHO-taught knowledge. Inclusion criteria for patients was 1) have chronic pain and be managed by the ECHO-participating PCP, 2) was prescribed opioids during the time frame of the study, and 3) not presented during ECHO sessions. Informed consent was obtained prior to each site visit. 1) Increased use of ECHO-taught pain and opioid management strategies 2) Feasibility of using chart reviews to evaluate PCP performance following a CME activity Results 25 (53%) patients were male and the average age was 59 (± 14) years. 24 (51%) patients had two or more pain diagnoses at baseline, with musculoskeletal pain being the most prevalent at 81%. 26 (55%) patients had comorbid mental health conditions and 13 (28%) had sleep disorders. Trends in results showed marginal, but non-significant, improvements in PCP performance after ECHO as indicated by increased use of pain and opioid management strategies. Conclusions Conducting chart reviews was a challenging method to assess provider performance. Future work to assess provider performance should include a qualitative component (in-depth interviews or focus groups) in order to complement the quantitative data and provide context for care and management decisions.