Related Experiment Video
Updated: Jan 10, 2026
10:39
Qualitative and Quantitative Validation of Tools with Rating Scales Aimed at Assessing the Quality of University Service-Learning
Published on: August 29, 2025
998
A Pilot Study on Generative Artificial Intelligence's Reliability in Qualitative Research Quality Appraisal Using
Hisba Shereefdeen1, Abhinand Thaivalappil2,3, Ian Young3
1University of Guelph, ON, Canada.
Summary
Generative artificial intelligence (genAI) tools show potential in qualitative research appraisal. ChatGPT demonstrated inconsistent agreement with human reviewers on appraisal checklists, particularly for interpretive elements.
Area of Science:
- Qualitative Research Methods
- Artificial Intelligence in Research
- Health Sciences Research
Background:
- Generative artificial intelligence (genAI) is increasingly integrated into research workflows.
- The application of genAI for quality appraisal in qualitative research is underexplored.
- Systematic review processes have begun exploring genAI, but quality appraisal needs further investigation.
Purpose of the Study:
- To evaluate ChatGPT's performance in appraising qualitative research using established checklists.
- To compare genAI appraisals with human assessments for consistency and reliability.
- To identify areas where genAI excels and struggles in qualitative research quality appraisal.
Main Methods:
- Two human reviewers applied the Critical Appraisal Skills Program (CASP) and Joanna Briggs Institute (JBI) checklists to 21 qualitative studies.
- ChatGPT (GPT-5) was prompted to appraise the same studies using the checklists.
- Interrater reliability (kappa) and crude agreement were calculated to compare human and genAI assessments.
Main Results:
- Agreement between human reviewers and ChatGPT ranged from none to moderate for CASP items (kappa: <.00-.46) and none to substantial for JBI items (kappa: <.00-.83).
- Highest agreement was observed for objective, reporting-based items (e.g., study aims, ethics, value, participant voices).
- Greatest disagreement occurred for subjective, interpretive items (e.g., research design, researcher-participant relationships, worldview-methodology congruity).
Conclusions:
- ChatGPT (GPT-5) reliably identifies objective components in qualitative research appraisal but inconsistently assesses nuanced, context-dependent elements.
- Current genAI use for qualitative research quality appraisal requires careful human oversight and adherence to transparency and data privacy principles.
- Further research is needed to refine genAI tools for complex qualitative appraisal tasks.
Related Concept Videos
01:29Reliability and Validity
13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
03:46Qualitative Analysis
23.6K
For solutions containing mixtures of different cations, the identity of each cation can be determined by qualitative analysis. This technique involves a series of selective precipitations with different chemical reagents, each reaction producing a characteristic precipitate for a specific group of cations. Metal ions within a group are further separated by varying the pH, heating the mixture to redissolve a precipitate, or adding other reagents to form complex ions.
For instance, group IV...
For instance, group IV...
23.6K
01:10Qualitative Analysis
1.2K
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
1.2K
01:19Quality Assurance
934
Quality assurance is the overarching term used to describe the activities employed to ensure the proper performance of a system. These activities can be classified into three categories: quality control, quality assessment, and internal corrective measures. Typically, these activities work cyclically: quality control is performed before and during the analysis, while quality assessment occurs during and after the investigation. Internal corrective measures are implemented based on the findings...
934
01:15Systematic Error: Methodological and Sampling Errors
8.5K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
8.5K
01:25Distribution Reliability and Automation
476
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
476