De-identification is not enough: a comparison between de-identified and synthetic clinical notes

Area of Science:

• Medical Informatics
• Data Privacy
• Artificial Intelligence

Background:

• De-identification is a common method for protecting privacy in sensitive clinical data.
• Synthetic data generation is emerging as a privacy-preserving alternative.
• Advancements in generative models prompt investigation into synthetic clinical notes for research.

Purpose of the Study:

• To assess the privacy efficacy of de-identified clinical notes against membership inference attacks.
• To develop a novel method for generating synthetic clinical notes using large language models.
• To evaluate the utility of synthetic clinical notes in clinical domain tasks and their privacy implications.

Main Methods:

• Demonstrated the vulnerability of de-identified clinical notes to membership inference attacks.
• Proposed a new approach for synthetic clinical note generation leveraging state-of-the-art large language models.
• Assessed the performance of generated synthetic notes on a clinical task.
• Developed a method to test privacy risks of models trained on synthetic data.

Main Results:

• De-identification failed to prevent membership inference attacks on real clinical notes.
• Synthetic clinical notes generated using large language models achieved performance comparable to real data.
• High performance of synthetic notes correlated with similar privacy concerns as real data.
• Membership inference attacks were feasible even when target models were trained on synthetic data.

Conclusions:

• De-identification alone is insufficient for robust privacy protection of clinical notes.
• Current state-of-the-art synthetic clinical notes, while useful for research tasks, inherit privacy risks.
• Further research is needed to explore synthetic data generation methods that offer improved privacy-utility trade-offs.