Assessing citation integrity in biomedical publications: corpus annotation and NLP models

Area of Science:

• Biomedical Informatics
• Computational Linguistics
• Scholarly Communication

Background:

• Citation accuracy is vital for scientific integrity and assessment.
• Quotation errors in citations can distort scientific evidence and are difficult for humans to detect.
• Automated methods are needed to identify citation inaccuracies in biomedical publications.

Purpose of the Study:

• To construct a corpus of biomedical citations with manually annotated accuracy.
• To develop and evaluate natural language processing (NLP) models for identifying citation quotation errors.
• To assess the performance of different NLP approaches, including generative large language models.

Main Methods:

• Manual annotation of 3063 citation instances from 100 highly-cited biomedical publications.
• Development of an NLP pipeline combining sentence retrieval (BM25 with MonoT5 reranker) and claim verification (MultiVerS model).
• Exploration of few-shot in-context learning using GPT-4 for citation accuracy classification.

Main Results:

• Approximately 39.18% of annotated citations contained accuracy errors.
• The best-performing NLP model achieved 0.59 micro-F1 and 0.52 macro-F1 scores.
• GPT-4 showed higher accuracy for correct citations but lower for erroneous ones compared to the fine-tuned model.

Conclusions:

• Citation quotation errors are subtle and challenging for current NLP models to detect.
• The developed NLP models show potential for improving citation quality and accuracy in scientific literature.
• The created corpus and NLP model are publicly available to facilitate further research.