使用人工智能来支持系统审查的方法质量评估
Manuel Marques-Cruz1, Filipe Pinto2, Rafael José Vieira3
1MEDCIDS - Department of Community Medicine, Information and Health Decision Sciences, Faculty of Medicine, University of Porto, Porto, Portugal; CINTESIS@RISE - Health Research Network, University of Porto, Porto, Portugal; Public Health Unit Marão e Douro Norte, Local Health Unit Trás-os-Montes e Alto Douro, Vila Real, Portugal.
Journal of clinical epidemiology
|August 27, 2025
概括
大型语言模型 (LLM) 可以有效地评估系统审查的质量. 一个微调的GPT-3.5模型显示了高准确性,显示了改进基于证据的指导方针的潜力.
科学领域:
- 医疗保健中的人工智能
- 证据综合
- 系统审查方法
背景情况:
- 系统性审查的多元化方法质量影响决策.
- 大型语言模型 (LLM) 提供了高效质量评估的潜力.
- 简化系统审查评估有助于制定指南.
研究的目的:
- 开发和评估基于LLM的工具来评估系统审查的方法质量.
- 确定LLM在这个任务中的准确性和可靠性.
主要方法:
- 在100个系统审查中评估了8个LLM (五个基础,三个微调).
- 使用27项验证的ReMarQ工具进行质量评估.
- 使用准确度,卡帕和F1分数对比LLM的表现.
主要成果:
- 一个微调的GPT-3.5模型实现了高性能 (精度为96.5%,kappa为0.90,F1得分为0.91).
- 在所有ReMarQ项目中,该模型的准确度超过了80%和0.60卡帕.
- 在表现最好的LLM重复评估中观察到很高的一致性.
结论:
- 在系统性审查的方法上,LLM 具有很大的潜力,可以准确地支持质量评估.
- 基于LLM的工具可以提高证据综合评估的效率和可靠性.
- 这项技术可以促进将高质量的证据纳入临床指南.
相关概念视频
Quality Assurance
198
Quality assurance is the overarching term used to describe the activities employed to ensure the proper performance of a system. These activities can be classified into three categories: quality control, quality assessment, and internal corrective measures. Typically, these activities work cyclically: quality control is performed before and during the analysis, while quality assessment occurs during and after the investigation. Internal corrective measures are implemented based on the findings...
198
Systematic Error: Methodological and Sampling Errors
2.3K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
2.3K
Data Validation
234
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Key parameters for method validation include:
234
Qualitative Analysis
678
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
There are two main approaches to qualitative analysis:...
678
The Scientific Method in Nursing Process
12.7K
The scientific method provides the foundation for any research. It is the most reliable and objective of all forms of gaining knowledge and guides in applying research-based evidence in practice and conducting future research.
When using research findings to change practice, one must understand the process used to guide a study. The scientific method is a systematic, step-by-step process that supports the data's validity, reliability, and generalizability. As a result, findings can be...
When using research findings to change practice, one must understand the process used to guide a study. The scientific method is a systematic, step-by-step process that supports the data's validity, reliability, and generalizability. As a result, findings can be...
12.7K
Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches
174
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
174


