Use of artificial intelligence to support the assessment of the methodological quality of systematic reviews
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
View abstract on PubMed
Summary
This summary is machine-generated.Large language models (LLMs) can efficiently assess systematic review quality. A fine-tuned GPT-3.5 model demonstrated high accuracy, showing potential for improving evidence-based guideline development.
Area Of Science
- Artificial Intelligence in Healthcare
- Evidence Synthesis
- Systematic Review Methodology
Background
- Heterogeneous methodological quality of systematic reviews impacts decision-making.
- Large language models (LLMs) offer potential for efficient quality assessment.
- Streamlining systematic review evaluation aids guideline development.
Purpose Of The Study
- To develop and evaluate an LLM-based tool for assessing systematic review methodological quality.
- To determine the accuracy and reliability of LLMs in this task.
Main Methods
- Assessed eight LLMs (five base, three fine-tuned) on 100 systematic reviews.
- Utilized the 27-item validated ReMarQ tool for quality assessment.
- Compared LLM performance against human reviewers using accuracy, kappa, and F1-score.
Main Results
- A fine-tuned GPT-3.5 model achieved high performance (accuracy 96.5%, kappa 0.90, F1-score 0.91).
- This model exceeded 80% accuracy and 0.60 kappa for all ReMarQ items.
- High consistency was observed in repeated assessments by the best-performing LLM.
Conclusions
- LLMs show significant potential for accurately supporting systematic review methodological quality assessment.
- LLM-based tools can enhance the efficiency and reliability of evidence synthesis evaluation.
- This technology can facilitate the incorporation of high-quality evidence into clinical guidelines.
These videos have been matched automatically. Contact us if they are not relevant.
These videos have been matched automatically. Contact us if they are not relevant.
Related Concept Videos
Quality assurance is the overarching term used to describe the activities employed to ensure the proper performance of a system. These activities can be classified into three categories: quality control, quality assessment, and internal corrective measures. Typically, these activities work cyclically: quality control is performed before and during the analysis, while quality assessment occurs during and after the investigation. Internal corrective measures are implemented based on the findings...
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Method validation is a crucial process in analytical chemistry designed to confirm that a given method consistently produces reliable and high-quality results. This process is essential when a method is applied to different sample matrices or when procedural modifications are made, ensuring that the results meet acceptable standards across various applications.
Key parameters for method validation include:
Specificity: The ability of the method to accurately measure the target analyte without...
Qualitative analysis is the process of identifying elements, ions, or compounds in an unknown sample. It is the first and most fundamental type of analysis based on the hierarchy of analytical goals. This hierarchy is significant as it provides a structured approach to scientific research, with qualitative analysis serving as the initial step, providing essential information before moving on to quantitative or other forms of analysis.
There are two main approaches to qualitative analysis:...
The scientific method provides the foundation for any research. It is the most reliable and objective of all forms of gaining knowledge and guides in applying research-based evidence in practice and conducting future research.
When using research findings to change practice, one must understand the process used to guide a study. The scientific method is a systematic, step-by-step process that supports the data's validity, reliability, and generalizability. As a result, findings can be...
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...