Lightweight Reinforcement Learning for Priority-Aware Spectrum Management in Vehicular IoT Networks

Area of Science:

• Computer Science
• Electrical Engineering
• Artificial Intelligence

Background:

• Vehicular Internet of Things (V-IoT) is crucial for intelligent transportation systems (ITSs), supporting diverse applications with strict Quality-of-Service (QoS) demands.
• Existing spectrum management techniques struggle with the dynamic nature of V-IoT networks and the computational demands of advanced solutions like deep reinforcement learning (DRL).
• There is a need for efficient, real-time spectrum management frameworks suitable for roadside unit (RSU) deployment in V-IoT.

Purpose of the Study:

• To propose a lightweight and interpretable reinforcement learning (RL)-based spectrum management framework for V-IoT networks.
• To introduce and evaluate two enhanced Q-Learning variants: VPADQ-C and Q-UCB, for improved spectrum allocation.
• To provide a baseline (Risk-Aware Heuristic) for comparing learning-driven approaches against traditional methods.

Main Methods:

• Developed Value-Prioritized Action Double Q-Learning with Constraints (VPADQ-C) using Constrained Markov Decision Process (CMDP) and online primal-dual optimization.
• Introduced contextual Q-Learning with Upper Confidence Bound (Q-UCB) incorporating uncertainty-aware exploration and Success-Rate Prior (SRP).
• Created a comprehensive simulation framework modeling traffic, fading, and energy dynamics to assess performance metrics.

Main Results:

• VPADQ-C demonstrated superior energy efficiency (≈8.425×10^7 bits/J) and over 60% reduction in interruption probability.
• Q-UCB achieved faster convergence (≈190 episodes), lowest blocking probability (≈0.0135), and minimal mean delay (≈0.351 ms).
• Both methods outperformed conventional Q-Learning and Double Q-Learning, maintaining fairness (≈0.364) and throughput (≈28 Mbps) with scalable training times.

Conclusions:

• The proposed RL-based framework offers a viable solution for real-time spectrum management in V-IoT.
• VPADQ-C and Q-UCB provide distinct advantages in energy efficiency, reliability, convergence speed, and latency.
• The frameworks are suitable for large-scale V-IoT deployments, meeting URLLC-grade requirements under dense vehicular traffic.