Using artificial intelligence to semi-automate trustworthiness assessment of randomized controlled trials: a case...

Area of Science:

• Medical Research Integrity
• Artificial Intelligence in Clinical Research
• Evidence-Based Medicine

Background:

• Randomized controlled trials (RCTs) are crucial for evidence-based medicine but can be compromised by fabricated data.
• Assessing RCT trustworthiness is vital to prevent misinformation in clinical guidelines and protect public health.
• Current methods for detecting problematic RCTs are inefficient and labor-intensive.

Purpose of the Study:

• To explore the potential of large language models (LLMs) in accelerating the assessment of RCT trustworthiness.
• To demonstrate the application of ChatGPT (GPT-4o) in evaluating an RCT using the TRACT checklist and automating data extraction.

Main Methods:

• A case study utilized ChatGPT to process an RCT PDF and apply the Trustworthiness in Randomised Controlled Trials (TRACT) checklist.
• The study focused on automating data table extraction for statistical analysis to enhance trustworthiness assessment.
• A step-by-step process was outlined, with considerations for future improvements.

Main Results:

• ChatGPT successfully completed assigned tasks, providing precise answers for TRACT checklist items and synthesizing information effectively.
• An 84% agreement rate was observed between ChatGPT and human assessment for TRACT items, significantly speeding up the qualitative review.
• ChatGPT accurately extracted and reorganized data tables, with three out of four achieving 100% accuracy, facilitating further analysis.

Conclusions:

• ChatGPT shows promise in semi-automating RCT trustworthiness assessment, though user prompting is currently required.
• Further development aims to improve accuracy and reduce user dependency for large-scale paper analysis.
• The ultimate goal is a fully automated process for evaluating numerous RCTs, which appears feasible based on initial findings.