Reinforcement Learning of Chaotic Systems Control in Partially Observable Environments

Area of Science:

• Fluid dynamics
• Control theory
• Machine learning

Background:

• Controlling chaotic systems is crucial in engineering, but real-world applications face partial observability due to limited sensing.
• Partial observability degrades control performance compared to full observability.
• The impact of memory types on controller performance in chaotic regimes remains poorly understood.

Purpose of the Study:

• Investigate reinforcement learning for controlling chaotic flows with partial observations.
• Evaluate performance loss with decreasing sensor availability.
• Compare recurrent neural networks (RNNs) with a novel transformer-based memory mechanism.

Main Methods:

• Utilized the Kuramoto-Sivashinsky equation with forcing as a model system.
• Tested control in various dynamic regimes, from mild to strong chaos.
• Implemented and compared RNN-based memory with a transformer-based attention mechanism.

Main Results:

• Performance degradation was quantified as sensor count decreased.
• The attention-based transformer framework demonstrated robust outperformance across different chaotic regimes.
• The novel mechanism showed improved control in highly chaotic environments.

Conclusions:

• Attention-based mechanisms, particularly transformers, are well-suited for controlling chaotic systems.
• This approach offers enhanced control in challenging, highly chaotic fluid dynamics.
• The findings advance the application of AI in complex dynamical system control.