A Contrastive Free Energy-Enhanced Transformer Framework for Efficient Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multiagent reinforcement learning (MARL) faces challenges with training efficiency and performance stability.
• Existing MARL algorithms often struggle to effectively integrate perception and decision-making processes.

Purpose of the Study:

• To propose a novel efficient MARL algorithm, contrastive free energy-enhanced multiagent Transformer (CFMAT).
• To enhance training efficiency and performance stability in MARL by integrating upstream perception and downstream decision-making.

Main Methods:

• Developed CFMAT, a joint training framework integrating representation encoder, prediction model, and Transformer-based decision-making module.
• Employed contrastive learning for unified observation representation and a novel contrastive free energy loss inspired by active inference (AIF) for upstream stability.
• Co-trained the representation encoder with an actor-critic network in the downstream module.

Main Results:

• CFMAT demonstrated significantly improved training efficiency compared to state-of-the-art baselines.
• Experimental results showed enhanced stable performance across various multiagent benchmark scenarios.
• The joint training framework facilitated efficient learning of the overall observation representation.

Conclusions:

• CFMAT effectively addresses key challenges in MARL, offering superior training efficiency and performance stability.
• The integration of upstream perception and downstream decision-making, guided by contrastive learning and AIF, is crucial for advanced MARL.
• CFMAT represents a significant advancement in developing robust and efficient multiagent systems.