ExpMRC: explainability evaluation for machine reading comprehension

Area of Science:

• Artificial Intelligence
• Natural Language Processing

Background:

• Pre-trained Language Models (PLMs) have achieved human-level performance on some Machine Reading Comprehension (MRC) tasks.
• Reliability in real-world applications necessitates MRC systems that provide not only answers but also explanations.

Purpose of the Study:

• Introduce ExpMRC, a novel benchmark for evaluating the textual explainability of MRC systems.
• ExpMRC includes four datasets (SQuAD, CMRC 2018, RACE+, C3) with added evidence annotations.

Main Methods:

• Develop baseline MRC systems using state-of-the-art PLMs.
• Employ unsupervised methods to extract answer and evidence spans, bypassing the need for human-annotated evidence.

Main Results:

• Experimental results indicate that current models perform significantly below human capabilities on the ExpMRC benchmark.
• The findings suggest ExpMRC presents a considerable challenge for existing explainable MRC systems.

Conclusions:

• ExpMRC serves as a challenging benchmark for assessing the explainability of MRC systems.
• The proposed benchmark and baseline systems are publicly available to facilitate further research.