Suppression of antiferromagnetic order by strain-enhanced frustration in honeycomb cobaltate

Layered honeycomb cobaltates are predicted as promising for realizing the Kitaev quantum spin liquid, a many-body quantum entangled ground state characterized by fractional excitations. However, they exhibit antiferromagnetic ordering at low temperatures, hindering the expected quantum state. We demonstrate that controlling the trigonal distortion of CoO₆ octahedra is crucial to suppress antiferromagnetic order through enhancing frustration in layered honeycomb cobaltates. Using heterostructure engineering on Cu₃Co₂SbO₆ thin films, we adjust the trigonal distortion of CoO₆ octahedra and the resulting trigonal crystal field. The original Néel temperature of 16 kelvin in bulk Cu₃Co₂SbO₆ decreases (increases) to 7.8 kelvin (22.7 kelvin) in strained Cu₃Co₂SbO₆ films by decreasing (increasing) the magnitude of the trigonal crystal fields. The first-principles calculation suggests the enhancement of geometrical frustration as the origin of the suppression of antiferromagnetism. This finding supports the potential of layered honeycomb cobaltate heterostructures and strain engineering in realizing extremely elusive quantum phases of matter.