A human-LLM collaborative annotation approach for screening articles on precision oncology randomized controlled...

Area of Science:

• Bibliometrics
• Artificial Intelligence in Research

Background:

• Supervised learning accelerates systematic review screening but requires extensive manual annotation.
• Large language models (LLMs) offer rapid screening but face reliability challenges.
• Developing efficient and reliable annotation methods is crucial for systematic reviews.

Purpose of the Study:

• To design an efficient and reliable human-LLM collaborative annotation method for article screening.
• To optimize LLM performance for high recall in identifying relevant articles.
• To validate the efficiency and reliability of the collaborative annotation approach.

Main Methods:

• A human-LLM collaborative annotation strategy focusing on verifying LLM-identified positive samples was developed.
• Prompts were iteratively optimized using a manually annotated dataset to achieve near-perfect LLM recall.
• The optimized LLM screened articles, followed by human verification of positive annotations, applied to precision oncology RCT screening.

Main Results:

• Iterative prompt optimization achieved near-perfect LLM recall (100% tuning, 85.71% validation).
• The collaborative method achieved an F1 score of 0.9583 and reduced annotation workload by ~80%.
• A BioBERT model trained on collaborative data outperformed one trained on LLM-only data, confirming reliability.

Conclusions:

• The human-LLM collaborative annotation method enhances efficiency and reliability in article screening.
• This approach offers significant support for conducting systematic reviews and meta-analyses.
• The findings highlight the potential of AI-assisted methods in scientific literature evaluation.