A dataset for evaluating clinical research claims in large language models.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Trial Analysis

Background:

• Large language models (LLMs) offer potential for verifying health claims.
• Challenges include LLM hallucination and logical statement comprehension in healthcare.
• Scrutiny of LLMs is essential for reliable clinical applications.

Purpose of the Study:

• Introduce CliniFact, a novel dataset for evaluating LLM performance in clinical research claim verification.
• Benchmark LLMs using hypothesis testing results from clinical research.
• Assess the capabilities of both discriminative and generative LLMs in this domain.

Main Methods:

• Constructed CliniFact from clinical trial data, including interventions, outcomes, and results.
• Linked derived claims to supporting evidence in scientific publications.
• Evaluated LLMs, including BioBERT and Llama3-70B, against the CliniFact dataset.

Main Results:

• CliniFact comprises 1,970 instances from 992 trials and 1,540 publications.
• Discriminative models achieved higher accuracy (e.g., BioBERT at 80.2%) compared to generative models (e.g., Llama3-70B at 53.6%).
• The difference in performance was statistically significant (p-value < 0.001).

Conclusions:

• CliniFact serves as a valuable benchmark for assessing LLM accuracy in clinical claim verification.
• Discriminative models demonstrate superior performance over generative models for this task.
• Further development is needed to improve LLM reliability in healthcare claim verification.