Automatic Generation of a Mechanical Properties Question-Answering Data Set for Language Model Benchmarking: A...

Area of Science:

• Materials Science
• Natural Language Processing
• Artificial Intelligence

Background:

• Contextualized language models (LMs) have potential for mining materials science information.
• Progress is hindered by the lack of domain-specific question-answering (QA) datasets.
• Existing QA benchmarks are often small and manually curated, limiting scalability.

Purpose of the Study:

• To introduce MechQA, a large-scale, automatically generated QA dataset for materials science.
• To provide a training resource for adapting LMs to the materials science domain.
• To evaluate the performance of fine-tuned transformer models using the MechQA dataset.

Main Methods:

• Automatically distilled 202,068 QA pairs on mechanical properties from 125,967 scientific articles.
• Covered five key mechanical properties: ultimate tensile strength, yield strength, fracture strength, Young's modulus, and ductility.
• Fine-tuned BERT-base, XLNet-base (extractive), and LLaMA-3.1-Instruct (generative) models using the MechQA dataset.

Main Results:

• Manual evaluation confirmed high quality of MechQA (83.76% precision, 89.09% recall, 86.34% F1 score).
• Fine-tuned extractive models (BERT, XLNet) achieved strong performance (e.g., BERT: 78.03% EM/84.50% F1) with improved calibration.
• The generative LLaMA model achieved competitive results (80.48% EM/86.25% F1), with extractive models showing strong performance despite smaller size.

Conclusions:

• MechQA serves as a valuable, large-scale resource for materials science QA.
• Automatic QA dataset generation is an effective data-centric method for domain adaptation of LMs.
• Fine-tuned transformer models demonstrate significant improvements in extracting materials science information.