Evaluating the accuracy of automated and semi-automated anonymization tools for unstructured health records

Utilization of unstructured clinical text in research is limited by the presence of protected health identifiers (PHI) within the text. To maintain patient privacy, PHI must be de-identified. The use of anonymization tools such as Microsoft Presidio and Philter has been recognized as a potential solution to the challenges of manual de-identification. Therefore, the primary objective of this study is to evaluate the accuracy and feasibility of using Microsoft Presidio and Philter in de-identifying unstructured clinical text.

A sample of 200 neurosurgical documents, temporally distributed across 10 years, was extracted. The data were processed by Microsoft Presidio and Philter. Each document was manually screened for the ground truth which was used as a reference point to evaluate the accuracy of each tool. Data analysis was conducted using Python.

A median of 8 PHI were manually de-identified per document. Both tools were individually capable of de-identifying a median of 6 PHI per document. Each tool de-identified PHI with an accuracy of 96%. Presidio demonstrated precision of 0.51 and a recall of 0.74, while Philter had precision and recall of 0.35 and 0.79, respectively.

The performance of each tool supports their use in anonymizing unstructured clinical text. Formatting variations between texts limited the performance of both tools. To conclude, further research is required to optimize the tools' output and assess the reliability in de-identifying diverse and previously unseen clinical text, thus allowing the use of unstructured clinical text in medical research.