对于离散时间马尔科夫跳跃系统的通用TD-Q学习控制方法
Jiwei Wen1, Huiwen Xue1, Xiaoli Luan1
1Key Laboratory of Advanced Process Control for Light Industry (Ministry of Education), School of Internet of Things Engineering, Jiangnan University, Wuxi 214122, China.
ISA transactions
|March 11, 2025
概括
本研究介绍了一种新的无模型时间差异Q (TD-Q) 学习方法,用于马尔科夫跳跃系统 (MJS) 中的强有力的控制. 这种方法确保了最佳的控制政策,即使未知系统动态和过渡概率.
科学领域:
- 控制理论 控制理论
- 机器学习 机器学习
- 随机系统 随机系统 随机系统
背景情况:
- 马尔科夫跳跃系统 (MJSs) 由于未知的动态和过渡概率 (TPs) 提出了强大的控制挑战.
- 现有的无模型方法往往难以同时处理MJS中的未知动态和TP.
研究的目的:
- 开发一种新的,无模型的时间差 Q (TD-Q) 学习方法,用于在离散时间 MJS 中进行强大的控制.
- 解决完全未知的动态和过渡概率的MJS.
- 提供一个全面的方法,包括未知动态的Q学习和未确定TP的TD学习.
主要方法:
- 提出了一个新的三元政策代框架,其中包括交替更新的动态循环.
- 该框架将 TD 价值函数与当前政策协调一致.
- 它使用TD值函数增强Q函数的矩阵内核 (QFMKs),并基于这些增强的QFMKs生成贪的策略.
主要成果:
- 经过足够的情节后,在代循环中证明了 TD 值函数,QFMK 和控制策略的最佳融合.
- 数字示例显示,与目前的MJS学习控制方法相比,有很大的好处.
- 使用对害虫的结构化种群动态模型验证实际适用性.
结论:
- 开发的TD-Q学习方法有效地实现了在MJS中具有未知的动态和TP的强有力的控制.
- 三元政策代框架确保了最佳的政策趋同.
- 该方法在现实场景中显示出显著的效率和实际适用性.
相关概念视频
01:24BIBO stability of continuous and discrete -time systems
318
System stability is a fundamental concept in signal processing, often assessed using convolution. For a system to be considered bounded-input bounded-output (BIBO) stable, any bounded input signal must produce a bounded output signal. A bounded input signal is one where the modulus does not exceed a certain constant at any point in time.
To determine the BIBO stability, the convolution integral is utilized when a bounded continuous-time input is applied to a Linear Time-Invariant (LTI) system....
To determine the BIBO stability, the convolution integral is utilized when a bounded continuous-time input is applied to a Linear Time-Invariant (LTI) system....
318
01:23Linear time-invariant Systems
200
A system is linear if it displays the characteristics of homogeneity and additivity, together termed the superposition property. This principle is fundamental in all linear systems. Linear time-invariant (LTI) systems include systems with linear elements and constant parameters.
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
200
01:26Feedback control systems
267
Feedback control systems are categorized in various ways based on their design, analysis, and signal types.
Linear feedback systems are theoretical models that simplify analysis and design. These systems operate under the principle that their output is directly proportional to their input within certain ranges. For instance, an amplifier in a control system behaves linearly as long as the input signal remains within a specific range. However, most physical systems exhibit inherent nonlinearity...
Linear feedback systems are theoretical models that simplify analysis and design. These systems operate under the principle that their output is directly proportional to their input within certain ranges. For instance, an amplifier in a control system behaves linearly as long as the input signal remains within a specific range. However, most physical systems exhibit inherent nonlinearity...
267
01:07Time-Domain Interpretation of PD Control
78
Proportional-Derivative (PD) control is a widely used control method in various engineering systems to enhance stability and performance. In a system with only proportional control, common issues include high maximum overshoot and oscillation, observed in both the error signal and its rate of change. This behavior can be divided into three distinct phases: initial overshoot, subsequent undershoot, and gradual stabilization.
Consider the example of control of motor torque. Initially, a positive...
Consider the example of control of motor torque. Initially, a positive...
78
01:21Transfer Function in Control Systems
280
The transfer function is a fundamental concept in the analysis and design of linear time-invariant (LTI) systems. It offers a concise way to understand how a system responds to different inputs in the frequency domain. It serves as a bridge between the time-domain differential equations that describe system dynamics and the frequency-domain representation that facilitates easier manipulation and analysis.
To derive the transfer function, consider a general nth-order linear time-invariant...
To derive the transfer function, consider a general nth-order linear time-invariant...
280
01:16Basic Discrete Time Signals
186
The unit step sequence is defined as 1 for zero and positive values of the integer n. This sequence can be graphically displayed using a set of eight sample points, showing a step function starting from n=0 and remaining constant thereafter.
The unit impulse or sample sequence is mathematically expressed as zero for all n values except at n=0, where it is one. The unit impulse sequence, denoted by δ(n), is the first difference of the unit step sequence, while the unit step sequence u(n) is...
The unit impulse or sample sequence is mathematically expressed as zero for all n values except at n=0, where it is one. The unit impulse sequence, denoted by δ(n), is the first difference of the unit step sequence, while the unit step sequence u(n) is...
186