SIC based RL for massive MIMO NOMA signal detection for different modulation schemes under diverse channel conditions

Massive-multiple input and Multiple Outputs Non orthogonal multiple access (M-MIMO-NOMA) systems require efficient signal detection techniques to mitigate interference and enhance spectral efficiency, especially under diverse channel conditions and varying modulation schemes. This study investigates the performance of the Successive Interference Cancellation with Reinforcement Learning (SIC-RL) detector compared to conventional methods, including the Minimum Mean Square Error (MMSE), Maximum Likelihood Detection (MLD), Approximate Message Passing (AMP), Gauss-Seidel (GS), Conjugate Gradient (CG), and zero-forcing equalizer (ZFE). The analysis was conducted for 16-QAM, 64- Quadrature Amplitude Modulation (QAM), and 256-QAM in Rayleigh fading channels with 10% error. The simulation results indicate that SIC-RL outperforms traditional detectors in terms of bit error rate (BER), power spectral density (PSD), and computational complexity. At a BER of 10⁻³, SIC-RL achieves an SNR 11.2 dB (512-QAM), 6.6 dB (256-QAM), 5 dB (64-QAM) and 5.8 dB (64-QAM with 10% channel error) as compared with conventional methods. The PSD analysis shows that SIC-RL exhibits a 35% and 20% lower spectral leakage compared to contemporary methods, ensuring better spectral efficiency for diverse channel conditions. In terms of computational complexity, SIC-RL achieves near-logarithmic growth with the number of antennas, significantly reducing the processing burden compared to MLD, which has exponential complexity. Although ZFE and CG are computationally efficient, they suffer from noise amplification and poor BER performance. GS and AMP balance complexity and performance but still fall short of SIC-RL gains. Overall, SIC-RL has emerged as an optimal solution for massive MIMO signal detection, achieving a superior trade-off between BER, PSD, and computational efficiency across diverse modulation schemes and channel conditions. Critically, SIC-RL achieves near-quadratic complexity [Formula: see text] contrasting the exponential complexity [Formula: see text] of MLD and the cubic complexity [Formula: see text] of MMSE and ZFE, making it scalable for large antenna arrays. Iterative methods such as AMP, GS, and CG achieve lower complexity, but suffer from convergence issues or degraded BER. Despite requiring initial training, SIC-RL provides a favorable trade-off between the detection accuracy and processing cost, positioning it as a computationally efficient and high-performance detector for next-generation MIMO-NOMA systems.