Evaluation of the Cochrane Consumers and Communication Group's systematic review priority-setting project

Health researchers and funders are increasingly consulting with stakeholders to set their research agendas but these activities are rarely evaluated. The Cochrane Consumers and Communication Group (CCCG) conducted a priority-setting project for systematic reviews in partnership with stakeholders (consumers/patients, health professionals, policy-makers and others). In this paper, we aim to describe our evaluation of the project's processes and outcomes.

We used a 10-element conceptual framework designed to evaluate processes (e.g. stakeholder engagement, use of explicit process) and outcomes (e.g. improved decision-making quality, stakeholder acceptance and understanding) of health priority-setting. Data sources included empirical data (feedback surveys, project documents and CCCG editorial policies) and CCCG staff reflections. Data were analysed using content analysis.

The project met three and partially met two of the process elements, for example, by engaging key stakeholders throughout the project and using pre-determined and transparent methods that offered multiple and meaningful ways to contribute. The project met three and partially met two of the outcome elements. Stakeholders were satisfied with and accepted the process and an additional six Cochrane Review titles aligned with stakeholder priorities are now being conducted in partnership with stakeholders. The project has also directly influenced the editorial work of CCCG, for example, by shifting its organisational focus towards coproduction, and indirectly influenced the work of Cochrane's prioritisation and coproduction activities. Some areas were identified as having room for improvement, for example, there was low participation by people from diverse backgrounds, stakeholders could contribute to most but not all project stages, and there was no formal way for stakeholders to appeal decisions at project end. In the 3 years since its completion, the Cochrane Reviews are nearing completion but none of the reviews have been published.

We demonstrated that our priority-setting methods were broadly in line with best practice and the project resulted in many positive outcomes beyond just identifying the top priorities for research. Our evaluation framework and recommendations for future evaluations may be of use to priority-setting researchers planning similar activities.