Use of deep learning-based NLP models for full-text data elements extraction for systematic literature review tasks

Area of Science:

• Health Economics and Outcomes Research (HEOR)
• Natural Language Processing (NLP)
• Machine Learning

Background:

• Systematic literature reviews (SLRs) are crucial for HEOR evidence synthesis but are labor-intensive.
• Previous work developed machine learning for identifying relevant publications.
• This study explores NLP for automated data extraction from scientific literature.

Purpose of the Study:

• To investigate the feasibility of using NLP for automated data element extraction in SLRs.
• To compare the performance of different NLP algorithms (CRF, LSTM, BERT) for this task.
• To provide annotated corpora for the NLP community.

Main Methods:

• Collected and annotated 239 full-text articles for 12 variables across three HEOR topics: HPV Prevalence, Pneumococcal Epidemiology, and Pneumococcal Economic Burden.
• Trained and evaluated Conditional Random Fields (CRF), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations from Transformers (BERT) models.
• Shared three annotated corpora publicly as a benchmark.

Main Results:

• Deep learning algorithms outperformed conventional machine learning for SLR data element extraction.
• LSTM models achieved superior micro-averaged F1 scores (0.890, 0.646, 0.615) across the three tasks.
• CRF models showed limited performance, and BERT did not yield expected improvements in this specific context.

Conclusions:

• Deep learning, particularly LSTM, demonstrates superior performance for automated data extraction in HEOR SLRs.
• LSTM models are recommended for deployment due to their performance, generalizability, scalability, and cost-effectiveness.
• The shared corpora will facilitate further NLP research in SLR evidence synthesis.