StudyTypeTeller-Large language models to automatically classify research study types for systematic reviews

Area of Science:

• Biomedical Informatics
• Natural Language Processing
• Systematic Review Methodology

Background:

• Systematic reviews face challenges due to increasing publication volumes, making manual screening labor-intensive.
• Generative large language models (LLMs) like GPT show potential for automating scientific study type classification.
• The performance of GPT models in biomedical study classification and comparison with BERT models is not well-established.

Purpose of the Study:

• To evaluate and compare the performance of GPT-3.5 and GPT-4 against BERT models for multi-class scientific study type classification.
• To develop and utilize a human-annotated corpus for training and evaluating NLP models in this task.
• To assess the impact of advanced prompting strategies on GPT model performance.

Main Methods:

• A human-annotated corpus of 2,645 PubMed titles and abstracts was created, covering 14 distinct study types.
• GPT-3.5, GPT-4, and established BERT models were fine-tuned and evaluated on this corpus.
• Performance was measured using F1-scores, comparing the models' accuracy in classifying study types.

Main Results:

• Fine-tuned BERT models consistently outperformed GPT models, achieving F1-scores above 0.8.
• GPT models achieved F1-scores around 0.6, with advanced prompting strategies showing minimal improvement.
• The study demonstrates BERT's superior performance in this specific multi-class classification task.

Conclusions:

• Smaller, fine-tuned BERT models are more effective than larger GPT models for classifying scientific study types in biomedical literature.
• Automated classification methods, despite current limitations of GPT, are promising for reducing screening workload in systematic reviews.
• The developed annotated corpus is a valuable resource for advancing NLP applications in evidence synthesis.