Vector modulation instability in gas-filled multi-pass cells

Multi-pass cells (MPCs) have emerged as versatile systems for nonlinear pulse compression supporting large single-stage compression factors. However, their performance is limited at extreme B-integrals by the onset of secondary nonlinear effects, which lead to the appearance of parasitic spectral peaks. In this Letter, we report the experimental observation of vector modulation instability (VMI) as a primary limiting mechanism in gas-filled MPCs operating at B-integrals >30 radians. The VMI signal is polarized orthogonally to the input beam and distinct from spectral features arising due to quasi-phase-matched four-wave mixing (QPM-FWM). We experimentally show that the VMI gain scales with accumulated B-integral, rather than with only gas density or peak power. Crucially, employing a circularly polarized input pulse suppresses VMI. This mitigation strategy, combined with dispersion management, is confirmed via simulations as a way for single-stage compression factors to exceed 40.