Systematic Performance Evaluation of Reinforcement Learning Algorithms Applied to Wastewater Treatment Control...

Area of Science:

• Environmental Engineering
• Water Treatment Technologies
• Artificial Intelligence in Environmental Science

Background:

• Activated sludge wastewater treatment involves complex, nonlinear processes that are challenging and energy-intensive to operate.
• Optimizing these systems for high treatment levels, including nutrient removal, is a significant research focus.
• Machine learning, particularly reinforcement learning, shows promise for improving control strategies.

Purpose of the Study:

• To evaluate the efficacy of four reinforcement learning algorithms for optimizing activated sludge wastewater treatment.
• To minimize energy consumption (aeration and pumping) while ensuring effluent compliance.
• To compare reinforcement learning control with existing domain-based strategies.

Main Methods:

• A novel interface was developed between process modeling software and a Python reinforcement learning environment.
• Four reinforcement learning algorithms were tested: deep Q-learning, proximal policy optimization, synchronous advantage actor critic, and twin delayed deep deterministic policy gradient (TD3).
• Performance was evaluated using the Benchmark Simulation Model No. 1 (BSM1).

Main Results:

• TD3 consistently achieved high control optimization, maintaining treatment requirements.
• TD3 reduced aeration and pumping energy by 14.3% compared to the BSM1 benchmark control.
• TD3 outperformed the ammonia-based aeration control strategy.

Conclusions:

• The TD3 algorithm demonstrates significant potential for energy savings in wastewater treatment.
• Reinforcement learning offers a viable alternative to traditional control strategies for activated sludge systems.
• Further research is needed to enhance the robustness of reinforcement learning implementations.