TWT: Textual white-box transformer for natural language understanding

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Deep learning models, despite advancements, suffer from a 'black-box' nature, hindering interpretability.
• Current methods for enhancing interpretability often neglect token relational constraints, limiting effectiveness in natural language understanding.
• Existing approaches may not fully capture the nuances of natural language features.

Purpose of the Study:

• To propose a novel interpretable deep learning model for natural language understanding tasks.
• To develop a 'white-box' transformer that explicitly considers token relational constraints.
• To improve the interpretability and performance of natural language understanding models.

Main Methods:

• Introduced the Textual White-box Transformer (TWT) with a focus on sparse rate reduction and token relational constraints.
• Employed a low-rank sparse embedding strategy (LSES) and label-interacted mapping mechanism (LMM) in the preprocessing layer.
• Utilized multi-head subspace self-attention (MSSA) and a token-conditioned iterative shrinkage-thresholding algorithm (T-ISTA) in the transformer layer.

Main Results:

• TWT demonstrated superior performance compared to state-of-the-art baselines on four text classification datasets.
• The method maintained consistent performance while ensuring simplicity and interpretability.
• Experimental results validated the effectiveness of LSES, LMM, MSSA, and T-ISTA in enhancing model interpretability and performance.

Conclusions:

• The proposed TWT offers a significant advancement in interpretable natural language understanding.
• TWT effectively addresses the limitations of existing methods by incorporating token relational constraints.
• The model's applicability extends to self-supervised pretraining for learning structured textual embeddings without explicit labels.