Polarization anisotropy in absorption enhanced modulation transfer spectroscopy and laser locking for Rb D₁ line

In a spin-exchange relaxation-free (SERF) atomic magnetometer, the pump and probe laser frequencies are tuned to the D₁ transition line of alkali metal atoms to enhance measurement sensitivity. The D₁ line, a non-cycling transition, exhibits a low absorption coefficient. This study theoretically analyzes the influence of different parameters on the absorption coefficient of the D₁ line and proposes a polarization-dependent approach to enhance it. A frequency stabilization method based on modulation transfer spectroscopy (MTS) with an enhanced absorption coefficient is introduced. Compared with the conventional method, optimizing the probe beam's polarization increases the absorption coefficient, raising the amplitude of the MTS signal. The results show that using a stronger MTS signal for frequency stabilization reduces the laser's frequency noise. Using heterodyne measurement, we characterize the linewidth-narrowing effect of MTS locking and evaluate the frequency stability of the locked laser. The linewidth of each laser is reduced from the free-running 2.2 MHz to 885 kHz after MTS stabilization. The Allan deviation measurements of the beat signal between two locked lasers are 2.6 × 10^-11 /τ. This work provides a stable optical frequency reference for SERF atomic magnetometers and offers potential applications in extracting spatial information from optically pumped atoms.