Multilevel Evaluation of ECHO Autism Programs: Development and Pilot Study of a Standardized Minimum Dataset

The rising prevalence of autism highlights the challenges faced by health care professionals, educators, and guardians in managing autism care. ECHO (Extension for Community Healthcare Outcomes) Autism Communities was established to train professionals in best practice autism care and diagnosis through regular virtual case-based learning sessions. As programmatic offerings expanded to include more professions and ongoing care after diagnosis, program measures were adapted to fit the desired learning outcomes and were adapted over time. As a result, consistent measures were not in place to assess any individual program longitudinally, and assessments of all programs collectively were even further limited. The global expansion of the ECHO Autism model led to an insistent need for standardized outcome measures to evaluate effectiveness.

In this study, a team consisting of ECHO Autism leaders with more than 20 years of combined knowledge of ECHO Autism programs reviewed existing measures to create a standard set of participant survey metrics, the ECHO Autism Communities minimum dataset (MDS). The study included a rapid 3-month pilot with a small 3-program dataset, followed by a 2-year pilot across 8 programs.

The study team collected 42 surveys from existing programs, organized all survey metrics, and refined them into a standardized set of metrics. The MDS was designed to measure priority information, set consistent scales and anchors, use inclusive language, standardize response choices, and determine units of analysis to transform data into actionable information. An early version of the MDS was translated into Spanish and implemented as a 3-month pilot across programs in the United States, Argentina, and Chile. Participant program-level data for the 2-year pilot were visualized and presented in PDF reports. Wilcoxon signed-rank tests of significance were performed on postsurvey self-efficacy data for programs and across all programs. An interactive dashboard was created for comparisons.

Three-month pilot data provided actionable insight so that adjustments were made before ongoing evaluation. The first 2 years of MDS implementation showed consistency in evaluation and significant improvements in self-efficacy across 8 programs. ECHO Autism leaders appreciated clear and simple data visualizations that showcased aggregate participant demographics, as well as changes in self-efficacy and barriers.

As a result of this study, ECHO Autism leaders can assess and compare program effectiveness, participant characteristics, and demonstrate the program's impact on professionals providing autism care. This study highlights the importance of consistent outcome measures for evaluation and enabling data-informed decisions for similar professional training programs for delivering autism best practices.