Artificial Intelligence to Support Qualitative Data Analysis: Promises, Approaches, Pitfalls

Area of Science:

• Computer Science
• Social Sciences
• Research Methodology

Background:

• Artificial intelligence (AI) has a long history of application in qualitative data analysis (QDA), spanning over 25 years.
• Recent advancements in AI, particularly large language models, offer new possibilities for supporting QDA tasks.
• However, the effective integration and limitations of current AI tools in QDA require careful examination.

Purpose of the Study:

• To investigate the capabilities and limitations of AI, specifically ChatGPT-4, in performing various qualitative data analysis tasks.
• To conduct a scoping review of existing literature on AI-supported QDA to understand current trends and methodologies.
• To examine the potential benefits, drawbacks, and ethical considerations associated with AI in qualitative data analysis.

Main Methods:

• Direct application of ChatGPT-4 to analyze narrative datasets, involving iterative prompt engineering to assess task success.
• A comprehensive scoping review of scholarly articles on AI-supported QDA published up to May 2024.
• Literature synthesis and expert examination of AI's role in QDA, including its historical context, current applications, and future implications.

Main Results:

• Initial attempts with ChatGPT-4 for thematic analysis and complex tasks were unsuccessful, though summarization and keyword counting improved with prompt refinement.
• The scoping review identified 130 articles, with a significant increase in publications in 2023-2024, highlighting diverse AI methods like unsupervised learning, thematic analysis, and sentiment analysis.
• AI has been utilized for various QDA approaches including coding, thematic analysis, discourse analysis, and handling large datasets, with methods like unsupervised learning being prevalent.

Conclusions:

• AI has a well-established history in supporting diverse QDA methods, with evolving technologies increasing accessibility.
• Current AI tools, while promising, have limitations in complex QDA tasks, necessitating "human in the loop" oversight and researcher understanding.
• The integration of AI in QDA presents both opportunities and challenges, requiring careful consideration of ethical implications, data privacy, and potential workforce impacts.