Pretty Darn Good Control: When are Approximate Solutions Better than Approximate Models

Area of Science:

• Ecology
• Computational Science
• Operations Research

Background:

• Optimal control methods struggle with real-world system complexities like dimensionality and data heterogeneity.
• Existing approaches often simplify models, raising questions about the optimality of solutions derived from approximate models.
• Advances in deep reinforcement learning (DRL) offer potential solutions for complex control problems.

Purpose of the Study:

• To investigate the application of DRL for approximating optimal control policies in complex ecological systems.
• To assess if DRL can derive effective management strategies without explicit system models.
• To compare DRL-derived policies against established methods in fishery management.

Main Methods:

• Utilized deep neural networks within a DRL framework.
• Applied DRL to a non-linear, three-variable fishery model.
• Trained the DRL agent to learn a control policy (policy function) without prior model inference.

Main Results:

• The DRL agent successfully approximated a control policy for the fishery model.
• The discovered DRL policy outperformed both constant escapement and constant mortality policies.
• The DRL policy exhibited characteristics of constant escapement, with values dynamically adjusted based on inter-species stock sizes.

Conclusions:

• DRL can effectively derive robust control policies for complex systems where traditional optimal control methods fall short.
• Model-free DRL approaches show promise for adaptive and effective management in ecological and other real-world applications.
• The study demonstrates the potential of DRL to address challenges posed by dimensionality, process error, and data heterogeneity in optimal control.