Integrating Large Language Models into Qualitative Methods in Health Services Research: A Proof-of-Concept Study

Qualitative methods are widely used in health services research to derive context-specific insights and depth of understanding. Manual coding, a foundational technique in rigorous qualitative analysis, is highly resource and time-intensive and difficult to scale. This is a particular challenge in health services research, where repeated rounds of interviews are common and rapid turnaround is often required. Natural Language Processing (NLP), specifically Large Language Models (LLMs), have shown potential to enhance efficiency in qualitative analysis. However, there is limited research providing guidance on how to integrate LLMs while maintaining rigor and trustworthiness. In this proof-of-concept study, we propose, apply, and evaluate an NLP-assisted coding method in a health services research setting.

We analyzed 22 interviews among public health officials, law enforcement, community organizers, and medical professionals at one California county to examine existing substance use service gaps and needs. A primarily deductive codebook was iteratively refined until two coders achieved an inter-coder reliability (ICR) > 0.95 and was applied to the transcripts using ATLAS.ti. We developed an NLP-assisted method that uses a semantic shift algorithm to segment transcripts which are then passed to GPT-4 for code assignment and explanation using the codebook and coding guidelines developed during the manual process. We evaluated the method with a quantitative assessment of agreement between human and NLP-assigned codes, a qualitative and quantitative soundness assessment by two reviewers, and a comparative efficiency analysis.

The NLP-assisted method had moderate agreement with human coding (modified pooled Cohen's Kappa = 0.66), and 71.8% of codes were rated as sound by reviewers. Sound codes were more often observed for high-certainty and straightforward codes, and when text chunks were semantically well defined. The NLP-assisted method had more difficulty with non-linear conversation and entity-dependent codes. Coding time was reduced significantly from ~40 hours for the traditional method to ~1 hour for the NLP-assisted method.

These findings suggest that LLMs can be effectively incorporated into qualitative processes while maintaining rigor if humans are embedded into the process. By maintaining a human-in-the-loop workflow, our methodology allows for researchers to maintain familiarity with the data, define the research question(s) and codebook, and determine if there are results that are not sound. By incorporating LLMs into the coding stage of the process, key limitations of traditional qualitative methods in health services research can be addressed, such as scalability, and resource and time limitations.