ScienceQA: a novel resource for question answering on scholarly articles

Area of Science:

• Natural Language Processing
• Information Retrieval
• Computational Linguistics

Background:

• Machine Reading Comprehension (MRC) is crucial for extracting information from scholarly articles, especially during information overload.
• Existing challenges in building MRC models stem from the scarcity of human-annotated datasets.
• Automated information extraction aids researchers, reviewers, and editors in understanding scientific content efficiently.

Purpose of the Study:

• To introduce a novel, large-scale dataset for question answering (QA) on scientific articles.
• To establish a benchmark for evaluating machine reading comprehension models on scientific texts.
• To facilitate advancements in automated information extraction from academic literature.

Main Methods:

• A semi-automated approach was used to create a dataset of over 100,000 human-annotated context-question-answer triples.
• A baseline QA model using Bidirectional Encoder Representations from Transformers (BERT) was implemented.
• Two advanced models were developed: one based on Science BERT (SciBERT) and another combining SciBERT with Bi-Directional Attention Flow (Bi-DAF).

Main Results:

• The SciBERT-based model achieved the highest performance, obtaining an F1 score of 75.46%.
• The developed dataset and baseline models provide a foundation for future research in scientific QA.
• The study demonstrates the effectiveness of specialized models like SciBERT for processing academic documents.

Conclusions:

• The newly created dataset significantly contributes to the field of scientific document processing and QA.
• The research opens new avenues for developing more sophisticated machine reading comprehension systems for scholarly articles.
• The availability of the dataset and code encourages further research and development in automated scientific information extraction.