Implementation of an RL-based cyberattack detector using VaR thresholding approach

The growing interconnection of control systems and network technologies has intensified concerns about securing Cyber-Physical Systems (CPSs) and Networked Control Systems (NCSs) against sophisticated cyber threats. This paper presents an experimentally validated, model-free, real-time cyberattack detection method for a Liquid-Level Control CPS (LLC-CPS), which relies exclusively on measured data and Reinforcement Learning (RL) concepts. By formulating the Q-function and value function in quadratic form, the method uses Q-learning to estimate a Bellman deviation sequence as a criterion for anomaly detection, without any explicit model of the system. To tackle the heavy-tailed nature of this residual, a Value-at-Risk (VaR) thresholding approach is adopted for the first time in this context, enabling separation of normal behavior from attack conditions even under non-Gaussian distributions. The effectiveness of this method is demonstrated through practical experiments targeting three major cyber threats: False Data Injection (FDI), Denial-of-Service (DoS), and learning-based Man-in-the-Middle (MitM) attacks. Despite challenges posed by the integrator-like dynamic of the system and physical input limits, the results show reliable detection performance across five evaluation metrics: False Alarm Rate (FAR), Missed Alarm Rate (MAR), Detection Delay (DD), precision, and F1-score.