Artificial Intelligence in Systematic Literature Reviews: Social Work Ethics, Application, and Feasibility

Area of Science:

• Medical research methodology
• Artificial intelligence in research
• Social work research

Background:

• Scoping reviews are crucial for synthesizing evidence, but are time-intensive.
• Generative AI offers potential to streamline research processes.
• Peer support specialists play a vital role in managing opioid use disorder (OUD).

Purpose of the Study:

• To compare thematic findings from AI tools with manual scoping review results.
• To evaluate the effectiveness of generative AI in identifying themes in medical research.
• To assess AI's role in streamlining time-intensive research processes.

Main Methods:

• A human-driven scoping review on peer support specialists for OUD was conducted.
• Thematic findings were compared against outputs from AI tools (NotebookLM, UTVERSE, Gemini).
• AI tools were prompted to identify themes solely from 15 uploaded peer-reviewed articles.

Main Results:

• AI models identified 53% to 80% of themes from the manual analysis.
• AI tools detected novel themes but also generated inaccurate or missed themes.
• Significant variability exists in generative AI performance for thematic analysis.

Conclusions:

• Human expert review is essential to validate AI-generated themes for accuracy and relevance.
• A hybrid approach combining AI with expert review can enhance research methodology.
• Establishing best practices for human-AI collaboration is crucial for social work research.