Retrieval augmented scientific claim verification

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Clinical Trial Data Analysis

Background:

• Automating the verification of scientific claims is crucial for evidence-based medicine.
• Existing methods often lack efficiency in processing large volumes of biomedical literature.
• The PICO framework provides a structured approach for evaluating clinical evidence.

Purpose of the Study:

• To develop and evaluate an automated system, CliVER, for scientific claim verification using PubMed abstracts.
• To leverage retrieval-augmented techniques for efficient information extraction and claim assessment.
• To create a specialized dataset (CoVERt) for training and validating the system.

Main Methods:

• Developed CliVER, a system integrating retrieval-augmented techniques, sentence extraction, and PICO framework analysis.
• Employed an ensemble of deep learning models for classifying claim support, refutation, or neutrality.
• Constructed the CoVERt dataset with 15 COVID-19 drug claims and 96 labeled clinical trial abstracts.
• Evaluated CliVER on CoVERt and the SciFact dataset for label prediction accuracy.

Main Results:

• CliVER achieved an F1 score of 0.92 on the CoVERt dataset for label prediction.
• The ensemble model surpassed individual state-of-the-art models by 3-11% in F1 score.
• Compared to clinicians, CliVER showed 79.0% precision in abstract retrieval, 67.4% in sentence selection, and 63.2% in label prediction.

Conclusions:

• CliVER shows promise in automating scientific claim verification by utilizing PubMed abstracts and retrieval-augmented strategies.
• The system's performance suggests a viable approach for harnessing clinical trial data for claim assessment.
• Further research is needed to explore CliVER's full clinical utility.