Human versus machine: A comparative analysis of qualitative coding by humans and ChatGPT-4

Area of Science:

• Psychology
• Artificial Intelligence
• Research Methodology

Background:

• Generative AI is increasingly impacting psychology training, practice, and research.
• Qualitative content analysis (QCA) is a key method in psychological research.

Purpose of the Study:

• To compare the procedures and products of QCA conducted by ChatGPT-4 and novice human researchers.
• To evaluate the advantages and disadvantages of AI-assisted versus human-led QCA in psychology.

Main Methods:

• Qualitative content analysis (QCA) was performed on open-ended survey responses from 60 school psychology trainers.
• ChatGPT-4 and novice human researchers independently conducted the QCA.
• Procedures (coding process) and products (codes, categories, themes) were compared.

Main Results:

• ChatGPT-4 produced QCA products similar to human coders and in significantly less time.
• ChatGPT-4's coding process lacked transparency, yielding some unclear codes and themes.
• Human coding provided a transparent, auditable, and systematic methodological approach, resulting in defensible themes.

Conclusions:

• AI tools like ChatGPT-4 offer efficiency gains in QCA for psychological research.
• Human oversight remains crucial for ensuring transparency, methodological coherence, and defensibility in qualitative analysis.
• Future research should explore hybrid approaches integrating AI and human expertise in qualitative research.