A residual-learning MMSE neural detector for 6G MIMO-OTFS systems under diverse channel conditions

Orthogonal time-frequency space modulation combined with multiple-input multiple-output transmission (MIMO-OTFS) has emerged as a strong waveform candidate for sixth-generation (6G) wireless networks because of its robustness against high mobility and doubly selective channels. However, reliable signal detection in large-scale MIMO-OTFS systems remains challenging owing to severe delay-Doppler coupling and channel state information (CSI), particularly under Rayleigh and Rician fading conditions. This paper proposes a residual-learning minimum mean square error neural detector (RL-MMSE-ND) to address these challenges under both perfect and imperfect CSI, including scenarios with up to a 20% channel estimation error. The proposed detector integrates a conventional minimum mean square error (MMSE) front-end with a lightweight residual-learning neural network that learns only the residual interference caused by CSI mismatch and delay-Doppler effects, thereby preserving MMSE stability while introducing minimal learning overhead. Extensive simulations were conducted for representative large-scale MIMO-OTFS configurations to evaluate the bit error rate (BER) versus signal-to-noise ratio (SNR), BER versus delay spread, power spectral density (PSD) characteristics, inference latency, and training convergence. Numerical results demonstrate that the proposed RL-MMSE-ND achieves 10-13 dB SNR gain at a BER of 10^-3 compared to zero-forcing equalization and MMSE detectors, while requiring 3-6 dB lower SNR than the maximum likelihood, QR decomposition-based M-algorithm detection, and deep learning-based detectors under both Rayleigh and Rician fading. Moreover, the proposed method reduces inference latency by over 60% compared to long short-term memory and bidirectional long short-term memory detectors and achieves 5-8 dB lower out-of-band emissions while adding only linear computational overhead beyond MMSE detection. These results confirmed the novelty and practical significance of the proposed approach for robust, low-latency, and low-complexity MIMO-OTFS detection in future 6G wireless systems.