Reinforcement Learning-Enhanced Botnet Defense System in Grid Topology Networks Using the SIRO Framework

Digitalizing essential services opens up a new risk of exposing critical infrastructure to botnet infections. In a grid topology network, the neighbor-to-neighbor paths can be used by the malicious botnet to spread the infection. Previous white-hat worm launchers used heuristics and supervised learning to exterminate botnets, which demand specific conditions or a suitable dataset to be effective. Although reinforcement learning addressed these issues, it requires a longer time to train. This article proposes a framework to shorten training and improve the effectiveness of reinforcement learning. The framework applies four key principles: (1) surveying the network status with multi-tensor input, (2) removing irrelevant actions via a novel Chebyshev-based masking strategy, (3) reinforcing key actions with rewards, and (4) optimizing rewards for winning. Four reinforcement learning algorithms are implemented to evaluate the framework, which are vanilla policy gradient, deep Q-network, proximal policy optimization, and MuZero in a stylized grid topology network simulation. An ablation study indicates that the masking used in identify accounts for the majority of the improvement, whereas multi-channel in Survey alone can reduce performance without complementary masking, rewards, and optimization. With the mean winning rate improved by 49.129% and mean win efficiency improved by 118.8031% against our previous work, the framework effectiveness is confirmed in stylized simulations.