在使用CASP和JBI检查清单进行定性研究质量评估时,对生成人工智能的可靠性进行试点研究
Hisba Shereefdeen1, Abhinand Thaivalappil2,3, Ian Young3
1University of Guelph, ON, Canada.
概括
生成型人工智能 (genAI) 工具在定性研究评估方面显示出潜力. 聊天GPT在评估检查清单上与人类审稿人达成不一致的协议,特别是在解释性元素方面.
科学领域:
- 定性研究方法 定性研究方法
- 研究中的人工智能.
- 卫生科学研究 卫生科学研究
背景情况:
- 生成型人工智能 (genAI) 越来越多地融入研究工作流程中.
- 在定性研究中,genAI用于质量评估的应用尚未得到充分研究.
- 系统审查过程已经开始探索genAI,但质量评估需要进一步调查.
研究的目的:
- 用已建立的检查清单来评估ChatGPT在评估定性研究方面的表现.
- 为了比较genAI评估与人类评估的一致性和可靠性.
- 确定genAI在定性研究质量评估中脱而出和扎的领域.
主要方法:
- 两个人体审查者将批判性评估技能计划 (CASP) 和乔安娜·布里格斯研究所 (JBI) 的检查清单应用于21项定性研究.
- 聊天GPT (GPT-5) 被要求使用检查清单来评估相同的研究.
- 测试者之间的可靠性 (kappa) 和粗略的一致性是为了比较人类和genAI的评估而计算的.
主要成果:
- 人类审查员和ChatGPT之间的协议在CASP项目中从没有到中度 (kappa: <.00-.46) 之间,在JBI项目中从没有到实质性 (kappa: <.00-.83).
- 对客观的,基于报告的项目 (例如,研究目标,伦理,价值,参与者的声音) 观察到最高的协议.
- 最大的分歧发生在主观的,解释性项目 (例如,研究设计,研究人员与参与者的关系,世界观与方法的一致性).
结论:
- 聊天GPT (GPT-5) 在定性研究评估中可靠地识别客观组件,但不一致地评估细微的,取决于上下文的元素.
- 目前用于定性研究质量评估的genAI使用需要仔细的人类监督和遵守透明度和数据隐私原则.
- 需要进一步的研究来完善genAI工具,以应对复杂的定性评估任务.
相关概念视频
Reliability and Validity
13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
Qualitative Analysis
23.6K
For solutions containing mixtures of different cations, the identity of each cation can be determined by qualitative analysis. This technique involves a series of selective precipitations with different chemical reagents, each reaction producing a characteristic precipitate for a specific group of cations. Metal ions within a group are further separated by varying the pH, heating the mixture to redissolve a precipitate, or adding other reagents to form complex ions.
For instance, group IV...
For instance, group IV...
23.6K
Qualitative Analysis
1.2K
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
1.2K
Quality Assurance
934
Quality assurance is the overarching term used to describe the activities employed to ensure the proper performance of a system. These activities can be classified into three categories: quality control, quality assessment, and internal corrective measures. Typically, these activities work cyclically: quality control is performed before and during the analysis, while quality assessment occurs during and after the investigation. Internal corrective measures are implemented based on the findings...
934
Systematic Error: Methodological and Sampling Errors
8.5K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
8.5K
Distribution Reliability and Automation
476
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
476

