Integrating Large Language Models into Thematic Analysis Workflows for Healthcare Research.

Area of Science:

• Health Informatics
• Qualitative Research Methods

Background:

• Thematic analysis is crucial for qualitative research but can be labor-intensive.
• Large language models (LLMs) offer potential for automating or assisting in qualitative data analysis.

Purpose of the Study:

• To evaluate the effectiveness of a lightweight, open-source LLM pipeline in accelerating thematic analysis.
• To compare LLM performance against manual coding in identifying concerns within a semi-structured interview.

Main Methods:

• A mixed-methods case study of Québec's electronic health-record rollout was used.
• Manual coders and four LLMs, including Hermes 3, applied a predefined codebook to a semi-structured interview.
• LLM extraction accuracy and validity of identified concern dimensions were assessed.

Main Results:

• Hermes 3 demonstrated high extraction accuracy, identifying 54 valid concern dimensions out of 70.
• Combined human-machine coding identified 67 unique valid concerns.
• LLMs uncovered 11 themes that were missed during manual coding.

Conclusions:

• LLMs can significantly reduce manual effort in thematic analysis and reveal overlooked themes.
• Expert validation remains essential for ensuring the rigor of LLM-assisted analysis.
• A hybrid workflow integrating LLM-driven extraction with researcher manual coding is recommended for optimal results.