Ghost poisoning: Making users invisible to speaker verification models

We demonstrate that speaker verification models are amenable to the creation of "ghost speakers." In particular, we devise a model poisoning method based on contrastive learning that causes the system to perform very poorly for a specific individual, while largely maintaining expected accuracy for all other users. This provides the ghost with a robust layer of identity obfuscation that could be exploited for malicious purposes. We test our method using three popular speaker verification models and demonstrate that it increases the equal error rate for the ghost by a factor of 10 or 20 times.