RNN-Bi-LSTM spectrum sensing algorithm for NOMA waveform with diverse channel conditions

Non-Orthogonal Multiple Access (NOMA) has emerged as a prominent technique for enhancing spectral efficiency in beyond-fifth-generation (5G) and sixth-generation (6G) wireless systems. However, its performance is highly dependent on channel conditions, necessitating robust spectrum-sensing methods. This study proposes a novel Recurrent Neural Network (RNN)-based Bidirectional Long Short-Term Memory (RNN-Bi-LSTM) model to enhance the spectral performance of NOMA under various channel conditions, particularly the Rician and Rayleigh fading channels. The proposed approach was evaluated using key performance metrics, including the probability of detection (PD), probability of false alarm (PFA), bit error rate (BER), and power spectral density (PSD). Simulation results show that RNN-Bi-LSTM achieves 100% PD at - 5 dB and - 2.5 dB SNR, outperforming conventional methods such as RNN (- 3 dB and - 1.5 dB), LSTM (- 1 dB and 0.3 dB), CSD (0.2 dB and 5.5 dB), matched filter (MF) (1 dB and 0.5 dB), and energy detection (ED) (2.3 dB and 2.7 dB) in Rician and Rayleigh channels, respectively. Additionally, the RNN-Bi-LSTM model shows a 23.36% improvement in PSD suppression under Rician conditions compared with Rayleigh conditions, reflecting the benefits of LoS-enhanced propagation in reducing spectral leakage and improving detection accuracy. BER performance also improves, achieving 10⁻⁵ at 8.8 dB and 5.8 dB SNR, whereas other methods require higher SNR. Furthermore, the model provides a more accurate PSD estimation, reduces spectral leakage, and enhances the spectrum utilization. Overall, RNN-Bi-LSTM demonstrated superior adaptability to varying channel conditions, making it a robust and efficient solution for NOMA-based spectrum sensing in advanced wireless communication systems.