Transformer model with external token memories and attention for PersonaChat

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Current dialog systems rely on large models and datasets, with information solely within the model structure.
• Existing approaches often struggle to efficiently and accurately mimic human conversational abilities.

Purpose of the Study:

• To introduce a novel transformer model with external token memory and attention (Tmema).
• To enhance dialog systems by enabling them to define and remember objects within a conversation, inspired by human cognition.

Main Methods:

• Developed Tmema, a transformer model incorporating external token memory and attention.
• Utilized a bidirectional self-attention mechanism and external memory in the encoder.
• Integrated latent information with external attention in the decoder during text generation.

Main Results:

• Tmema demonstrated superior performance compared to state-of-the-art methods on the PersonaChat dataset.
• The model achieved high scores in both automatic and human evaluations.
• The proposed architecture effectively enhances dialog system capabilities.

Conclusions:

• Tmema offers a significant advancement in developing human-like dialog systems.
• The external memory mechanism is key to improving conversational AI accuracy and efficiency.
• The model's open-source availability facilitates further research and development in conversational AI.