An Improved Math Word Problem (MWP) Model Using Unified Pretrained Language Model (UniLM) for Pretraining.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Natural Language Understanding (NLU) and Generation (NLG) are crucial for text processing but often require extensive training data.
• Traditional NLU/NLG models like Bag of Words and N-Grams face limitations with data scarcity.
• Pretraining has emerged as a vital technique to overcome data dependency in language models.

Purpose of the Study:

• To propose a semi-supervised method for Math Word Problem (MWP) tasks using unsupervised pretraining and supervised tuning.
• To leverage the Unified pretrained Language Model (UniLM) for improved performance in MWP tasks.
• To reduce the data requirements for training NLU/NLG models by utilizing pre-learned parameters.

Main Methods:

• A semi-supervised approach combining unsupervised pretraining and supervised fine-tuning based on the UniLM framework.
• Initialization of new task models with parameters from previously learned tasks to avoid training from scratch.
• Integration of Autoregressive (AR) and Autoencoding (AE) language model advantages for enhanced prediction capabilities (one-way, sequence-to-sequence, two-way).

Main Results:

• The proposed model achieved a maximum accuracy of 79.57% on MWP tasks with over 20,000 mathematical questions.
• Demonstrated superior performance compared to traditional models in math word problem-solving.
• Identified that incorrect arithmetic order significantly impacts the generation of correct solution expressions.

Conclusions:

• The UniLM-based semi-supervised method effectively addresses data scarcity in MWP tasks.
• Transfer learning through pretraining significantly enhances model performance and reduces data requirements.
• Accurate prediction of arithmetic order is critical for generating correct mathematical solutions.