Efficient Privacy-Budgeted Large Language Model for Sensitive Healthcare.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Healthcare Informatics

Background:

• Large language models (LLMs) show promise for healthcare question answering.
• Adapting LLMs with sensitive clinical data risks unintended memorization and privacy leakage.

Purpose of the Study:

• To present MedDP, an efficient privacy-budgeted adaptation framework for LLMs in healthcare.
• To deliver an auditable utility-privacy trade-off under differential privacy.
• To mitigate memorization and privacy leakage in LLM adaptation.

Main Methods:

• MedDP applies differential privacy by stochastic gradient descent (DP-SGD) to low-rank adapter parameters, freezing the base model.
• It utilizes per-example gradient clipping and calibrated Gaussian noise for explicit privacy accounting with low overhead.
• Introduces Adaptive Clipping (AC) for optimization stability and Budget-Aware (BA) checkpoint selection to optimize utility within a privacy budget.

Main Results:

• Experiments on MedQA and MedMCQA benchmarks demonstrate a clear privacy-utility frontier.
• Privacy-budgeted training mitigates memorization signals while maintaining stable performance.
• Non-private adaptation improves accuracy but increases memorization.

Conclusions:

• MedDP offers an efficient and auditable solution for privacy-preserving LLM adaptation in healthcare.
• The framework achieves a favorable balance between utility and privacy, crucial for sensitive data applications.
• Canary negative log-likelihood (NLL) proves a sensitive metric for evaluating memorization in LLMs.