Human Analysis vs. Artificial Intelligence: Analyzing of Qualitative Medical Students' Narratives

Area of Science:

• Medical Education
• Artificial Intelligence
• Qualitative Research

Background:

• Reflective narratives from medical students offer rich qualitative data on experiences.
• Manual analysis of these narratives is time-consuming and costly.
• Artificial intelligence (AI) tools, specifically large language models (LLMs), show potential for streamlining analysis.

Purpose of the Study:

• To explore the efficacy of LLMs in analyzing medical student narratives.
• To compare LLM analysis with manual qualitative analysis.
• To assess LLM performance in contextual and thematic analysis of sensitive medical encounter narratives.

Main Methods:

• Utilized LLMs to analyze semi-structured narratives from medical students reflecting on bad news delivery.
• Compared LLM-derived analysis against a previously conducted manual analysis.
• Evaluated performance across contextual (demographics) and thematic categories.

Main Results:

• LLMs successfully processed narrative data and adhered to formatting requirements.
• LLM performance varied; strong in straightforward categories (e.g., age), weaker in nuanced aspects (e.g., surprise).
• Narrative ambiguities and complexities presented challenges for LLM interpretive accuracy.

Conclusions:

• LLMs can enhance the efficiency of narrative analysis by extracting explicit contextual information.
• LLMs show promise as a complementary tool for human qualitative researchers.
• Further research is needed to refine LLM capabilities for complex narrative interpretation and synergistic human-AI analysis.