Reinforcement Learning-Enhanced Botnet Defense System in Grid Topology Networks Using the SIRO Framework

Area of Science:

• Cybersecurity
• Artificial Intelligence
• Network Security

Background:

• Digitalization of essential services increases critical infrastructure vulnerability to botnet infections.
• Malicious bots spread infections through neighbor-to-neighbor paths in grid topology networks.
• Existing botnet extermination methods (heuristics, supervised learning, reinforcement learning) have limitations.

Purpose of the Study:

• To propose a novel framework to shorten reinforcement learning (RL) training time and enhance its effectiveness for botnet detection.
• To improve the defense against botnet infections in critical infrastructure networks.

Main Methods:

• A framework incorporating multi-tensor network status surveying, Chebyshev-based action masking, reward reinforcement for key actions, and reward optimization for winning.
• Implementation and evaluation of four RL algorithms (vanilla policy gradient, deep Q-network, proximal policy optimization, MuZero) within a grid topology network simulation.
• Ablation study to determine the impact of individual framework components.

Main Results:

• The Chebyshev-based masking strategy significantly contributed to performance improvement.
• Multi-tensor surveying alone, without masking and reward optimization, could decrease performance.
• The framework demonstrated a 49.129% increase in mean winning rate and a 118.8031% increase in mean win efficiency compared to previous work.

Conclusions:

• The proposed framework effectively shortens RL training and improves botnet detection efficacy.
• The combination of multi-tensor input, action masking, and reward optimization is crucial for enhancing RL performance in cybersecurity applications.
• The framework offers a promising solution for defending critical infrastructure against botnet threats.