Angular instability in high optical power suspended cavities

Advanced gravitational wave detectors use suspended test masses to form optical resonant cavities for enhancing the detector sensitivity. These cavities store hundreds of kilowatts of coherent light and even higher optical power for future detectors. With such high optical power, the radiation pressure effect inside the cavity creates a sufficiently strong coupling between test masses whose dynamics are significantly altered. The dynamics of two independent nearly free masses become a coupled mechanical resonator system. The transfer function of the local control system used for controlling the test masses is modified by the radiation pressure effect. The changes in the transfer function of the local control systems can result in a new type of angular instability which occurs at only 1.3% of the Sidles-Sigg instability threshold power. We report the experimental results on a 74 m suspended cavity with a few kilowatts of circulating power, for which the power to mass ratio is comparable to the current Advanced LIGO. The radiation pressure effect on the test masses behaves like an additional optical feedback with respect to the local angular control, potentially making the mirror control system unstable. When the local angular control system is optimised for maximum stability margin, the instability threshold power increases from 4 kW to 29 kW. The system behaviour is consistent with our simulation, and the power dependent evolution of both the cavity soft and hard mode is observed. We show that this phenomenon is likely to significantly affect the proposed gravitational wave detectors that require very high optical power.