Comparison of Qualitative Analyses Conducted by Artificial Intelligence Versus Traditional Methods

Area of Science:

• Health professions education research
• Qualitative analysis methodologies
• Artificial intelligence in research

Background:

• Qualitative research is underutilized in health professions education due to training gaps and time-intensive analysis.
• Generative artificial intelligence (AI), specifically large language models like GPT-4, presents opportunities to streamline qualitative research.
• The accuracy of AI-generated codes and themes in this field requires further investigation.

Purpose of the Study:

• To characterize qualitative analyses assisted by GPT-4 compared to traditional human analysis in health professions education.
• To evaluate the accuracy, alignment, relevance, and appropriateness of GPT-4-generated codebooks and themes.

Main Methods:

• Two health professions datasets were reanalyzed using GPT-4 after initial human analysis.
• GPT-4's outputs (codebooks and themes) were compared against prior human findings.
• Dichotomous ratings and collaborative discussions were used to assess GPT-4's strengths and weaknesses.

Main Results:

• GPT-4 analysis of 36 survey responses and seven interview transcripts showed general alignment with human-identified concepts.
• GPT-4 struggled to detect low-frequency codes and establish coherent code relationships.
• Nuance in theme descriptions and quote selection by GPT-4 was limited.

Conclusions:

• GPT-4 can serve as a supportive tool for human-led qualitative analysis but cannot replace it.
• Researchers need a solid understanding of qualitative research and the dataset to effectively utilize AI tools and identify limitations.