On the same (evaluation) page: a novel approach to enhance mixed-methods implementation evaluation.

Area of Science:

• Health Services Research
• Implementation Science
• Mixed Methods Research

Background:

• Quality improvement and implementation science evaluations often use complex mixed methods approaches with interdisciplinary teams.
• The COVID-19 pandemic and shift to remote work introduced significant challenges to these evaluations.
• Existing tools were insufficient for rigorous evaluation planning with geographically distributed, remotely working teams.

Purpose of the Study:

• To create a novel, replicable, seven-step process for developing a rigorous evaluation plan for a large, hybrid implementation-evaluation trial.
• To address the challenges of conducting complex evaluations with remotely working, interdisciplinary teams.

Main Methods:

• A seven-step process was developed: 1) select template and point person, 2) initial development, 3) asynchronous feedback, 4) gap analysis, 5) virtual synchronous discussion, 6) finalize document, 7) apply and solicit feedback.
• The process was applied to a large hybrid type III implementation-evaluation trial across nine medical centers.
• Incorporated team science approaches to enhance evaluation plan development.

Main Results:

• The novel seven-step evaluation plan tool enhanced the rigor and execution of a large program evaluation.
• The process facilitated clear communication, well-ordered workflow, and rigorous operationalization of evaluation aims.
• The evaluation plan development process served as a convening function, enhancing coordination among remote team members.

Conclusions:

• A structured, seven-step process can improve the rigor and cohesion of mixed methods evaluations for interdisciplinary teams, particularly those working remotely.
• This novel approach provides a replicable framework for developing robust evaluation plans in complex implementation science projects.
• The developed process supports team coordination and effective operationalization of evaluation goals in challenging research environments.