Related Experiment Video
Updated: May 8, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large language models for deductive qualitative content analysis in dementia-focused embedded pragmatic clinical
Jeffrey Turner1, Spencer Phillips Hey2, Zachary G Baker3
1Center for Gerontology and Healthcare Research, Brown University, 121 South Main Street, Suite 6, Providence, 02903, RI, USA. jeffrey_turner@brown.edu.
Implementation Science Communications
|May 7, 2026
Summary
Large language models (LLMs) automate thematic coding for intervention implementation studies, significantly reducing time and cost. This AI approach accelerates evaluations in embedded pragmatic clinical trials (ePCTs) while maintaining high accuracy.
Area of Science:
- Implementation Science
- Artificial Intelligence
- Dementia Care Research
Background:
- Thematic coding is crucial for characterizing intervention implementation in embedded pragmatic clinical trials (ePCTs), especially for dementia interventions.
- Manual coding is labor-intensive, hindering timely evaluations.
- Large language models (LLMs) offer potential for automating qualitative data analysis in implementation science.
Purpose of the Study:
- To develop and evaluate an automated algorithm using LLMs (Chat GPT-4o and GPT-4o-mini) for thematic coding of interview transcripts.
- To assess the performance of LLMs compared to human coders in identifying implementation determinants, barriers, and facilitators.
- To quantify the impact of LLM-powered workflows on efficiency (time and cost) in implementation research.
Main Methods:
- A Python-based system was developed to process semi-structured interview transcripts using LLMs.
- The system matched transcript excerpts to an established codebook, undergoing multiple refinement iterations with expert review.
- Performance was compared against human coding, evaluating accuracy, code type matching, and missed excerpts.
Main Results:
- The LLM consistently coded more excerpts than human coders, achieving up to 72.6% matching rates on individual transcripts.
- GPT-4o demonstrated superior performance over GPT-4o-mini, with higher matching rates for descriptive codes (89% vs. 69%).
- The LLM workflow resulted in a 97% reduction in time and a 99% reduction in cost per transcript.
Conclusions:
- LLM-powered thematic analysis aligns well with human coding, offering a reliable supplementary tool for implementation science.
- While human oversight is necessary due to error rates, LLMs significantly enhance efficiency and scalability in qualitative research.
- This automated approach streamlines the analysis of implementation processes, potentially accelerating research and minimizing resource allocation.
Related Concept Videos
01:10Qualitative Analysis
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
03:46Qualitative Analysis
For solutions containing mixtures of different cations, the identity of each cation can be determined by qualitative analysis. This technique involves a series of selective precipitations with different chemical reagents, each reaction producing a characteristic precipitate for a specific group of cations. Metal ions within a group are further separated by varying the pH, heating the mixture to redissolve a precipitate, or adding other reagents to form complex ions.
For instance, group IV...
For instance, group IV...
01:30Dementia
Dementia is a collective term for cognitive disorders primarily affecting memory, thinking, and reasoning. It is not a specific disease but a syndrome, with Alzheimer's disease being the most common cause, accounting for approximately 60-80% of cases. Other types include vascular dementia, Lewy body dementia, and frontotemporal dementia. Dementia affects millions worldwide, particularly older adults, though it is not a normal part of aging.
The progression of dementia is generally gradual.
The progression of dementia is generally gradual.
01:22Dementia l: Introduction
Dementia is an acquired, progressive syndrome characterized by a decline in multiple cognitive domains severe enough to impair daily functioning and reduce independence. Although memory loss is a central feature, the diagnosis requires additional deficits involving language, executive function, visuospatial skills, judgment, calculation, or abstract reasoning. These cognitive impairments reflect underlying neurodegenerative or vascular processes that gradually disrupt neuronal networks...