CiteSure: retrieval-augmented large language models for faithful biomedical citation recommendation

Accurate citation of relevant publications is essential for scientific integrity in biomedical research. Large language models (LLMs) excel at text generation but often hallucinate fabricated or inaccurate citations. Retrieval-augmented generation (RAG) can mitigate these errors, yet current approaches lack semantic precision in evidence retrieval. This study aims to develop a domain-specific RAG system for reliable, context-specific biomedical citation recommendations.

We introduce CiteSure, a sentence-level citation recommendation tool designed to deliver reliable, evidence-based, and context-specific references using LLMs. CiteSure utilizes a 2-stage retrieval-augmented generation (RAG) framework, combining a domain-specific dense retriever (BioLLM2Vec) and reranker (BioRankLLaMA), adapted from LLaMA3-8B-Instruct using biomedical-specific training data. CiteSure leverages the complementary strengths of retrieval and generative LLM models, ensuring factual precision and contextual alignment. We evaluated CiteSure on a curated Alzheimer's disease dataset, comparing it to standalone LLMs and traditional retrieval-based methods.

CiteSure achieved 100% factual accuracy and the highest relevance score of 77.50%, outperforming all baselines. BioLLM2Vec retrieved relevant articles with over 80% accuracy in the top 100 candidates. BioRankLLaMA consistently outperformed baseline rerankers across MAP, MRR, and Precision@5 metrics, confirming the benefit of domain-specific adaptation and contrastive fine-tuning.

Our results demonstrate that CiteSure, built on a 2-stage retrieval-augmented generation framework, effectively integrates domain-specific retrieval with LLM-based generation to achieve substantial improvements over baseline approaches. Our work underscores the importance of domain-specific adaptation in biomedical citation recommendation and provides publicly available datasets, models, and code for support future research.