Is Multiclass Automatic Text De-Identification Worth the Effort?

Area of Science:

• Natural Language Processing
• Clinical Informatics
• Machine Learning

Background:

• Automatic de-identification of clinical text is crucial for protecting patient privacy by removing protected health information (PHI).
• Models for de-identification can be binary, replacing PHI with a generic tag, or multiclass, retaining specific PHI categories.
• Binary models are simpler but may reduce text informativeness, while multiclass models are more complex.

Purpose of the Study:

• To investigate whether the added complexity of multiclass de-identification models for clinical text justifies their development.
• To compare the accuracy and document readability of binary versus multiclass de-identification approaches.

Main Methods:

• Utilized the 2014 i2b2 dataset for model training and evaluation.
• Developed one binary and two multiclass de-identification models using the Conditional Random Field (CRF) machine learning algorithm.
• Assessed model accuracy (recall, precision, f-score) and performance metrics (training/testing time, memory).
• Evaluated document readability through a review by three independent raters, calculating inter-rater agreement using Kappa statistic.

Main Results:

• The multiclass model showed no significant accuracy advantage over the binary model, exhibiting lower recall (-1.9%) and only a marginal increase in precision (+0.6%).
• Multiclass models required additional computational resources compared to binary models.
• Reviewers achieved high inter-rater agreement (Kappa = 0.975), concluding that both binary and multiclass de-identification methods have a similar impact on document readability.

Conclusions:

• The findings suggest that the development of sophisticated multiclass classification for PHI may not offer substantial benefits in terms of system accuracy or output utility.
• Simpler binary de-identification models appear to be a more efficient and equally effective approach for clinical text anonymization.