Concerns of AI use in evidence synthesis based practices: collective views from the community

BACKGROUND: The use of artificial intelligence (AI) in research has become one of the most hotly debated topics. This is particularly true for the field of evidence synthesis where automation through AI may lead to substantial time and resource savings. Many researchers see the potential benefits of using AI technologies, yet there is hesitation around embedding AI in practice. We explored the concerns of those working in the field of evidence synthesis through a series of online and in-person events. METHODS: Data collection was conducted across two in-person and 2 online events: the Evidence Synthesis Hackathon (ESH) 2024, the Community, Opportunities, Research and Experience Information Retrieval (CORE) Forum, a Systematic Review Conversations (SRC) online seminar, and an online Horizon Scanning (HS) Survey. Inductive and deductive coding was utilised to synthesis data into broad themes and subthemes, independently for each event. A vote counting and ranking approach was used to triangulate data across events to capture convergent and divergent themes between participant groups. RESULTS: Across the four events we acquired a total of 248 data points (from 80 respondents) and responses were broadly similar across cohorts. Through synthesis and triangulation, we identified 10 overarching themes. The most prominent themes were knowledge and skills, and data management, respectively. Skills loss, skills gap and job loss were highlighted within the knowledge and skills theme. Bias, confidentiality and reliability were prominent for data management. Lower ranking concerns included environment, economics, AI market and costs. CONCLUSIONS: These are valid apprehensions faced by researchers across the field of evidence synthesis and should be considered in the broader discussion of AI. Development of rigorous methodologies and guidance may help to overcome these issues by facilitating responsible and transparent use of AI.