Tunable Quantum Magnetism in Frustrated Yb-Based van der Waals Materials

The border between magnetic order and a quantum disordered ground state is notoriously difficult to reach by design. Here we show that systematic anion substitution in a family of rare-earth van der Waals (vdW) magnets, YbChX (Ch = S, Se; X = Br, I), provides a direct chemical handle on this boundary. By replacing chalcogenide and halide ligands on the FeOCl-type lattice, we systematically tune the crystal electric field hierarchy, the degree of XY anisotropy, and the strength of quantum fluctuations in the ground-state Kramers doublet, affecting the exchange interactions without changing the distorted triangular lattice topology. The result is a series of Jeff = 1/2 magnets with large frustration factors (f = 9-76), extended short-range correlation regimes, and suppressed long-range order, placing the entire family in close proximity to magnetic quantum criticality. Far-infrared magnetospectroscopy, diffuse reflectance, and photoluminescence reveal that the same ligand substitution that reshapes the magnetic landscape simultaneously tunes f-f optical transitions, opening a route to access potential spin-photon coupling through chemical modifications. These results establish anion substitution as a powerful design principle for chemically navigating frustrated quantum magnets toward exotic ground states in the vdW family.