Reinforcement Learning Based Optimal Tracking Control Under Unmeasurable Disturbances With Application to HVAC...

Area of Science:

• Control Engineering
• Artificial Intelligence
• Machine Learning

Background:

• Traditional reinforcement learning (RL) control methods often require direct measurement or manipulation of disturbances, which is impractical in many real-world scenarios.
• Unmeasurable disturbances and unknown system dynamics pose significant challenges for achieving optimal control and accurate estimates.
• Existing RL techniques can suffer from bias in control estimates when disturbances are not directly accounted for.

Purpose of the Study:

• To develop an optimal controller using reinforcement learning (RL) that can effectively manage unmeasurable disturbances and unknown system dynamics.
• To eliminate the need for direct measurement or manipulation of disturbances in RL-based control systems.
• To prevent disturbance-induced bias in optimal control estimates.

Main Methods:

• A novel Q-learning framework is proposed, integrating a bias compensation mechanism with integral action.
• A bias-compensated Q-learning scheme learns disturbance-induced bias terms separately from optimal control parameters.
• State feedback and output feedback algorithms are developed using policy iteration (PI) and value iteration (VI).

Main Results:

• The proposed method successfully removes the need to measure or manipulate disturbances.
• The bias compensation mechanism prevents disturbance-induced bias in optimal control estimates.
• Algorithms guarantee the convergence of tracking error to zero, even with unmeasurable disturbances.
• The approach was validated on a practical heating, ventilating, and air conditioning (HVAC) zone controller.

Conclusions:

• The developed RL-based optimal controller effectively handles unmeasurable disturbances and unknown dynamics.
• The integration of bias compensation and integral action in Q-learning provides a robust solution for practical control problems.
• The methodology ensures reliable and accurate control performance, demonstrated through a real-world HVAC application.