Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches01:23

Types of Biopharmaceutical Studies: Controlled and Non-Controlled Approaches

407
Biopharmaceutical studies constitute a vital field aiming to enhance drug delivery methods and refine therapeutic approaches, drawing upon diverse interdisciplinary knowledge. In research methodologies, the choice between controlled and non-controlled studies significantly influences the study's reliability and accuracy.
Non-controlled studies, commonly employed for initial exploration, lack a control group, rendering them susceptible to biases and external influences. In contrast,...
407

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Anti-Malignancy and Anti-Inflammation Properties of Emodin, an Essential Constituent in Popular Laxative Botanicals.

The American journal of Chinese medicine·2026
Same author

Artificial Intelligence-Enabled Cardiac Function Estimation from Phone Videos of Echocardiograms.

medRxiv : the preprint server for health sciences·2026
Same author

Interpretable spatial multi-omics data integration and dimensionality reduction with SpaMV.

Nature communications·2026
Same author

Reporting Interest-Holder Engagement in Practice Guidelines: The RIGHT-MuSE Checklist.

Annals of internal medicine·2026
Same author

Amino acid-based biological age clock and its implications for human health and aging.

Nature communications·2026
Same author

Integrating Planetary Health in Health Guidelines (GRADE Guidance 46).

Annals of internal medicine·2026

Related Experiment Video

Updated: Jan 16, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.0K

Using Large Language Models to Assess the Consistency of Randomized Controlled Trials on AI Interventions With

Xufei Luo1,2,3,4,5, Zeming Li6, Zhenhua Yang7

  • 1Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, 199 Donggang West Road, Chengguan District, Lanzhou, 730000, China, 86 13893104140.

Journal of Medical Internet Research
|September 26, 2025
PubMed
Summary

Large language models (LLMs) show promise in evaluating research consistency. GPT-4 variants performed best in assessing artificial intelligence (AI) randomized controlled trials (RCTs) against CONSORT-AI standards, though human oversight remains crucial.

Keywords:
CONSORT-AIChatGPTartificial intelligencelarge language modelrandomized controlled trials

More Related Videos

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

1.3K

Related Experiment Videos

Last Updated: Jan 16, 2026

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

1.0K
Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
05:47

Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems

Published on: June 13, 2025

1.3K

Area of Science:

  • Artificial Intelligence in Medical Research
  • Clinical Trial Reporting Standards
  • Natural Language Processing Applications

Background:

  • Large language models (LLMs) demonstrate potential for evaluating research consistency.
  • Previous studies utilized LLMs to assess randomized controlled trial (RCT) abstracts against CONSORT-Abstract guidelines.
  • The consistency of AI-interventional RCTs adhering to CONSORT-AI standards, when evaluated by LLMs, is not well-established.

Purpose of the Study:

  • To evaluate the consistency of AI-interventional RCTs with CONSORT-AI standards using LLM-based chatbots.
  • To identify the performance of different LLM models in assessing adherence to CONSORT-AI guidelines.

Main Methods:

  • A cross-sectional study involving 6 LLM models to assess 41 RCTs on AI interventions from JAMA Network Open.
  • Queries were submitted via API with a temperature setting of 0 for deterministic responses.
  • Overall Consistency Score (OCS), recall, inter-rater reliability, and content consistency were analyzed, with independent verification of LLM responses.

Main Results:

  • GPT-4 variants exhibited the highest average OCS, with gpt-4-0125-preview achieving 86.5% (JAMA authors) and 81.6% (study authors).
  • GPT-3.5-turbo-0125 showed the lowest average OCS (61.9% and 63.0%).
  • CONSORT-AI Item 2 ('State the inclusion and exclusion criteria at the level of the input data') had the poorest evaluation (48.8% OCS), while Items 1, 5, 8, and 9 exceeded 80% OCS.

Conclusions:

  • GPT-4 variants show strong capability in assessing RCT consistency with CONSORT-AI.
  • Prompt refinement is necessary to improve the precision and consistency of LLM-based evaluations.
  • Human supervision and expertise are essential for reliable AI-driven assessments in medical research, enhancing evaluation efficiency and quality.