Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Transesophageal echocardiography and pulmonary artery catheter use and short-term outcomes in patients undergoing coronary artery bypass surgery: A multi-center retrospective study.

BMC cardiovascular disorders·2026
Same author

Comparing the Efficacy and Safety of Intra-articular Injection Treatments for Hip Osteoarthritis: A Systematic Review and Network Meta-analysis.

Orthopaedic journal of sports medicine·2026
Same author

A validity evaluation of lexicon-based sentiment analysis of medical students' clinical performance from in-training evaluation reports.

Medical education·2025
Same author

Development of a Transfer Learning-Based, Multimodal Neural Network for Identifying Malignant Dermatological Lesions From Smartphone Images.

Cancer informatics·2025
Same author

So You've Got a High AUC, Now What? An Overview of Important Considerations when Bringing Machine-Learning Models from Computer to Bedside.

Medical decision making : an international journal of the Society for Medical Decision Making·2025
Same author

Propofol-associated Hypertriglyceridemia: Development and Multicenter Validation of a Machine-Learning-Based Prediction Tool.

Journal of intensive care medicine·2025

Related Experiment Video

Updated: Jul 5, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

587

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study.

Eddie Guo1, Mehul Gupta1, Jiawen Deng2

  • 1Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.

Journal of Medical Internet Research
|January 12, 2024
PubMed
Summary
This summary is machine-generated.

Large language models like GPT-4 can significantly improve clinical research by efficiently screening titles and abstracts, aiding researchers and enhancing review accuracy. This AI-powered approach streamlines systematic reviews, saving time and resources.

Keywords:
Chat GPTGPTGPT-4LLMNLPabstract screeningclassificationextractextractionfree textlanguage modellarge language modelsnatural language processingnonopiod analgesiareview methodologyreview methodsscreeningsystematicsystematic reviewunstructured data

More Related Videos

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
07:50

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts

Published on: September 20, 2018

15.9K
Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
09:20

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications

Published on: February 23, 2019

8.7K

Related Experiment Videos

Last Updated: Jul 5, 2025

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
03:14

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

587
A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts
07:50

A Metadata Extraction Approach for Clinical Case Reports to Enable Advanced Understanding of Biomedical Concepts

Published on: September 20, 2018

15.9K
Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
09:20

Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications

Published on: February 23, 2019

8.7K

Area of Science:

  • Medical Informatics
  • Artificial Intelligence in Healthcare
  • Clinical Research Methodology

Background:

  • Systematic reviews of clinical research are time-intensive, relying on manual screening of thousands of titles and abstracts.
  • Accuracy and efficiency in this process are crucial for high-quality reviews and informed healthcare decisions.
  • Traditional human-powered screening requires substantial time and resources.

Purpose of the Study:

  • To evaluate the performance of OpenAI's generative pretrained transformer (GPT) and GPT-4 APIs for identifying relevant clinical research titles and abstracts.
  • To compare the AI models' accuracy and efficiency against human reviewers using real-world clinical review datasets.
  • To assess the potential of large language models in streamlining the systematic review process.

Main Methods:

  • A novel workflow was developed utilizing Chat GPT and GPT-4 APIs for screening clinical review titles and abstracts.
  • A Python script automated API calls with natural language screening criteria and a corpus of human-filtered data.
  • Performance was compared against human reviewers across 6 review papers, screening over 24,000 titles and abstracts.

Main Results:

  • The AI models achieved an accuracy of 0.91 and a macro F1-score of 0.60.
  • Sensitivity for excluded papers was 0.91, and for included papers, it was 0.76.
  • The AI demonstrated strong agreement with human consensus (κ=0.96) and could provide and correct reasoning for its classifications.

Conclusions:

  • Large language models, such as GPT-4, show significant potential to enhance the efficiency of clinical review processes.
  • These AI models can serve as valuable aids to researchers, streamlining workflows and saving time.
  • By improving efficiency and accuracy, AI can contribute to more reliable conclusions in medical research.