Comparing AI-Assisted Coding and Traditional Qualitative Analysis: a Study Examining Differences in Methods and...

Area of Science:

• Public Health
• Prevention Science
• Computational Social Science

Background:

• Qualitative data is crucial for understanding community health and behavior but is time-intensive to analyze.
• Traditional qualitative analysis requires extensive training and resources.
• Advancements in artificial intelligence (AI) offer potential solutions for efficient qualitative data coding.

Purpose of the Study:

• To compare the effectiveness of AI-assisted qualitative analysis with traditional content analysis.
• To evaluate similarities and differences in methods and results between AI and human coders.
• To assess the utility of AI in processing large volumes of qualitative data for public health.

Main Methods:

• Collected 2820 community comments from 43 events across 27 zip codes for a city/county food plan.
• AI-assisted analysis utilized transcription apps, GPT-4 Plus, and GPT for Sheets with public health oversight.
• Traditional analysis involved two trained coders, codebook development, reliability testing, and full content coding.

Main Results:

• AI-assisted and traditional methods yielded similar results for deductive coding (key food system aspects).
• Inductive coding results showed less comparability between AI-assisted and traditional methods.
• Both approaches used deductive and inductive codes to capture food system data.

Conclusions:

• AI-assisted qualitative analysis, with appropriate oversight, can enhance the processing of large qualitative datasets in prevention science.
• AI methods show potential for strengthening public health research by improving efficiency.
• Further refinement is needed to ensure comparable results for inductive coding using AI.