Evaluating Language Models for Biomedical Fact-Checking: A Benchmark Dataset for Cancer Variant Interpretation...

Area of Science:

• Genomic medicine
• Bioinformatics
• Computational biology

Background:

• Accurate genomic variant interpretation is crucial for precision oncology but is hindered by slow, expertise-dependent manual review processes.
• Public knowledgebases like CIViC curate variant interpretations but face bottlenecks in verification and review.

Purpose of the Study:

• To develop a benchmark dataset and automated pipeline (CIViC-Fact) for testing systems that verify cancer variant claims.
• To assess the efficiency and accuracy of AI models in biomedical fact-checking for knowledgebase curation.

Main Methods:

• Created CIViC-Fact, linking structured claims to sentence-level evidence from full-text articles, including expert annotations.
• Evaluated proprietary and open-source language models, fine-tuning an open-source model on CIViC-Fact.
• Applied the fact-checking pipeline to real CIViC entries to assess AI-assisted triage effectiveness.

Main Results:

• A fine-tuned open-source language model achieved the highest accuracy (89%) in verifying cancer variant claims.
• The fact-checking pipeline identified that reviewing less than 20% of content, focusing on flagged entries, could detect over half of all errors.
• AI-assisted triage significantly accelerates the review process while maintaining expert oversight.

Conclusions:

• CIViC-Fact provides a robust framework for biomedical fact-checking and improving knowledgebase curation.
• AI-assisted review can enhance the efficiency of curators by focusing expert attention on critical entries.
• This approach supports more rigorous and efficient knowledgebase curation in precision oncology.