Artificial Intelligence in Systematic Literature Reviews: Social Work Ethics, Application, and Feasibility

This study explores the alignment between themes identified by Artificial Intelligence (AI)-powered tools and those from a traditional, manual scoping review, focusing on generative AI's role in streamlining time-intensive research processes.

Thematic findings from a human-driven scoping review on peer support specialists in medical settings for opioid use disorder (OUD) were compared with outputs from NotebookLM, UTVERSE, and Gemini. Fifteen peer-reviewed articles were uploaded to each AI tool, and a standardized prompt directed the generative AI to identify themes using only the provided articles, which were then compared to the human-coded findings.

The AI models identified between 53% and 80% of the themes found in the original manual analysis. While AI tools identified novel themes that could broaden the scope of analysis, they also generated inaccurate or misleading themes and overlooked others entirely.

The variability in generative AI performance highlights its potential and limitations in thematic analysis. AI identified additional themes and misinterpreted or missed others. Human expert review remains necessary to validate the accuracy and relevance of generative AI, while addressing ethical considerations in alignment with the values of the social work profession.

A hybrid approach that combines generative AI with expert review has the potential to support current manual research approaches and establish a robust methodology. Continued evaluation, addressing limitations, and establishing best practices for human-AI collaboration and transparent reporting are crucial for the social work research field.