Comparing Human Insight and AI in Thematic Analysis of Nursing Education Reflections

Area of Science:

• Health Professions Education
• Artificial Intelligence in Education
• Qualitative Research Methods

Background:

• Interprofessional education (IPE) advancement is hindered by resource-intensive student learning evaluations.
• Reflective assignments in IPE encourage self-examination of learning and future interprofessional collaboration.
• Manual analysis of reflective assignments is time-consuming, limiting scalability.

Purpose of the Study:

• To compare human-led coding and theming with large language model (LLM) generated outputs for evaluating interprofessional reflections.
• To assess the efficacy and limitations of LLMs in analyzing qualitative data from IPE reflective assignments.

Main Methods:

• Qualitative descriptive study design.
• Comparison of traditional human coding and theming with LLM-generated coding and theming.
• Evaluation focused on interprofessional reflections from students.

Main Results:

• LLM-generated outputs for reflexive thematic qualitative analysis are not currently identical to those produced by human teams.
• LLMs demonstrate potential for efficient review of large datasets in educational research.

Conclusions:

• LLMs offer a method for efficient large-scale data review in educational contexts.
• Current LLMs have limitations and should not solely replace human analysis.
• Integrating LLMs with other evaluation methods is recommended for comprehensive assessment of interprofessional reflections.