Transformer-Based Nonlinear Blind Source Separation for Anti-Jamming in DSSS Satellite Communications

High-power jamming may drive the radio-frequency (RF) front end of a satellite receiver into a nonlinear regime, thereby invalidating the linear superposition assumption underlying conventional excision and blanking methods. We formulate dual-receiver direct-sequence spread-spectrum (DSSS) anti-jamming as a nonlinear source-separation problem in complex baseband using stacked I/Q observations. We then propose a time-domain separator that jointly estimates the desired DSSS signal and the jammer on a designated reference receiver. The separator combines a multi-scale convolutional front end with a Transformer encoder and is pretrained on synthetic nonlinear mixtures that include multi-tone or burst jamming as well as typical satellite impairments, including Doppler/carrier-frequency offset (CFO), phase noise, multipath, and additive white Gaussian noise (AWGN). Robustness under high-jammer-to-signal-ratio (JSR) conditions is improved through high-JSR oversampling and JSR-aware loss reweighting. After Stage I supervised pretraining on labeled synthetic mixtures, an optional Stage II mixture-only adaptation step further refines the separator using nonlinear reconstruction consistency and lightweight communication-motivated priors. Across 1000 test mixtures with JSRs from -5 to 15 dB, SNRs from 15 to 25 dB, and cubic coefficients a∈[0,0.5], the proposed method improves the desired-signal scale-invariant signal-to-noise ratio (SI-SNR) from -4.79 dB for the mixture baseline to 13.32 dB after supervised pretraining and to 17.73 dB after mixture-only blind fine-tuning. Over the same test set, the failure rate (SI-SNR < 0 dB) decreases from 60.7% to 2.3%.