Related Experiment Video
Updated: Jan 15, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1.0K
Evaluation of large language models within GenAI in qualitative research
Supriya D Mehta1,2, Souvik Paul3, Enid Awiti4
1Division of Infectious Diseases, Department of Medicine, Rush University College of Medicine, Chicago, USA. Supriyad@uic.edu.
Scientific Reports
|October 7, 2025
Summary
Large language models (LLMs) show potential for qualitative research but struggle with thematic analysis accuracy. Current AI tools can aid theme identification but are not yet reliable for rigorous qualitative research.
Area of Science:
- Qualitative Research Methods
- Artificial Intelligence in Health Research
Background:
- Large language models (LLMs) offer advanced capabilities for text analysis and response generation.
- Generative AI (GenAI) incorporating LLMs presents potential utilities for qualitative research applications.
- The COVID-19 pandemic significantly impacted sexual and reproductive health among adolescent girls and young women (AGYW).
Purpose of the Study:
- To compare thematic analyses from GenAI (GPT-4o) with human analysis in a qualitative study.
- To evaluate theme identification, quote selection, and quote quality between GenAI and human coders.
- To explore quantitative and qualitative sentiment analysis performed by GenAI.
Main Methods:
- A qualitative study on the COVID-19 pandemic's impact on AGYW's sexual and reproductive health in rural Kenya.
- LLM (GPT-4o) was used for thematic and sentiment analysis after transcript input and pre-processing.
- Two investigators independently reviewed GenAI outputs against qualitative research standards using a rubric.
Main Results:
- GenAI identified relevant sub-themes but some did not meet the threshold for a full theme compared to human analysis.
- GenAI exhibited low and variable performance in selecting supportive quotes for thematic and sentiment analysis.
- GenAI produced hallucinations and identified biases related to training data and cultural context limitations.
Conclusions:
- GenAI (GPT-4o) is currently unable to produce thematic analysis indistinguishable from human researchers.
- GenAI can serve as an aid for identifying themes, keywords, and basic narratives in qualitative data.
- Further advancements are needed for GenAI to reliably conduct rigorous qualitative thematic analysis independently.
Related Concept Videos
01:10Qualitative Analysis
1.3K
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
1.3K
03:46Qualitative Analysis
23.6K
For solutions containing mixtures of different cations, the identity of each cation can be determined by qualitative analysis. This technique involves a series of selective precipitations with different chemical reagents, each reaction producing a characteristic precipitate for a specific group of cations. Metal ions within a group are further separated by varying the pH, heating the mixture to redissolve a precipitate, or adding other reagents to form complex ions.
For instance, group IV...
For instance, group IV...
23.6K
01:22Language Development
848
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
848
01:11Genetic Lingo
113.8K
Overview
113.8K
01:30Typical Model Studies
619
Fluid mechanics model studies often utilize scaled-down systems to predict fluid behavior in full-scale environments, such as river flows, dam spillways, and structures interacting with open surfaces. Maintaining Froude number similarity in river models is crucial, as it replicates surface flow features like wave patterns and velocities.
619
01:27Language and Cognition
711
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
711