Integrating Large Language Models into Qualitative Methods in Health Services Research: A Proof-of-Concept Study.

Area of Science:

• Health Services Research
• Qualitative Analysis
• Natural Language Processing (NLP)

Background:

• Qualitative methods are crucial for in-depth health services research but are time-intensive.
• Manual coding presents scalability and efficiency challenges, especially with large datasets.
• Large Language Models (LLMs) offer potential for enhancing qualitative analysis efficiency.

Purpose of the Study:

• To propose, apply, and evaluate an NLP-assisted coding method for health services research.
• To assess the rigor, trustworthiness, and efficiency of integrating LLMs into qualitative analysis.
• To address the limitations of traditional manual coding in terms of time and resources.

Main Methods:

• Developed an NLP-assisted method using GPT-4 for code assignment and explanation.
• Applied a semantic shift algorithm to segment transcripts for NLP processing.
• Evaluated the method through human-NLP code agreement, soundness assessment, and efficiency analysis.

Main Results:

• The NLP-assisted method showed moderate agreement with human coding (Kappa = 0.66).
• 71.8% of NLP-assigned codes were rated as sound by reviewers.
• Coding time was drastically reduced from approximately 40 hours to 1 hour.

Conclusions:

• LLMs can be effectively integrated into qualitative analysis with a human-in-the-loop workflow.
• This approach maintains rigor and trustworthiness while addressing scalability and time constraints.
• The methodology empowers researchers to retain data familiarity and ensure result soundness.