Unsupervised utility evaluation of text anonymization methods via neural language models

Text anonymization methods strive to find a balance between privacy protection and utility preservation, where the latter refers to the fact that the anonymized documents are still analytically useful for research or business tasks. The performance of these methods is evaluated empirically, by comparing their outputs with human-based anonymizations through the standard precision and recall metrics. Whereas recall is used as a proxy for the level of privacy attained, precision is interpreted as the amount of utility preserved. Nonetheless, these metrics were not designed for evaluating privacy-oriented tasks and present several drawbacks. First, they assume a unique ground truth whereas, in text anonymization, several masking choices can be equally valid to prevent re-identification. Second, the human annotations used as ground truth are inherently subjective and prone to errors. Third, both metrics weight terms uniformly, thereby ignoring the varied impact that the terms' semantics have on utility and re-identification risk. To overcome these limitations, in this paper we present the first unsupervised utility metric for anonymized texts. Our metric relies on neural language models to quantify the utility loss incurred by anonymization. Empirical experiments on document clustering show that our metric captures the actual utility of the anonymized outcomes more accurately than precision, while being more sensitive to varying anonymization intensities. Together with a previously proposed privacy metric, our proposal defines a complete evaluation framework for text anonymization that does not require costly human annotations. Using this framework, we also report a comprehensive evaluation of a variety of text anonymization methods.