Inductive thematic analysis of healthcare qualitative interviews using open-source large language models: How does it...

Area of Science:

• Artificial Intelligence in Healthcare
• Qualitative Research Methodologies

Background:

• Large language models (LLMs) are gaining attention for clinical and research applications.
• Limited analysis exists on LLM utility in qualitative thematic analysis compared to human coders.
• No published studies have evaluated LLMs on real-world protected health information.

Purpose of the Study:

• To compare LLM performance against standard human thematic analysis.
• To analyze LLM use in real-world, semi-structured interviews within a psychiatric setting.

Main Methods:

• Utilized a 70 billion parameter open-source LLM on local hardware.
• Employed advanced prompt engineering techniques for theme generation.
• Applied three evaluation methods to quantify theme similarity between LLM and human analysis.

Main Results:

• LLM-generated themes demonstrated moderate to substantial similarity to human-generated themes.
• Jaccard similarity coefficients ranged from 0.44 to 0.69.
• These results indicate promising preliminary findings for LLM application.

Conclusions:

• Open-source LLMs can effectively generate robust themes from qualitative data.
• LLMs achieve substantial similarity to human-generated themes in thematic analysis.
• LLM validation in thematic analysis can enhance and democratize qualitative research.