Efficient Privacy-Budgeted Large Language Model for Sensitive Healthcare

Large language models (LLMs) are promising for healthcare question answering, yet adapting them with sensitive clinical data can induce unintended memorization and privacy leakage. We present MedDP, an efficient privacy-budgeted adaptation framework that delivers an auditable utility-privacy trade-off under ($\epsilon,\delta$)-differential privacy. Our method, MedDP, applies DP-SGD only to low-rank adapter parameters while freezing the base model, using per-example gradient clipping and calibrated Gaussian noise to enable explicit privacy accounting with minimal compute and memory overhead. To improve the stability of noisy optimization, we introduce Adaptive Clipping (AC) that updates the clipping bound online via a quantile-based rule with momentum. To avoid spending additional privacy budget on late-stage accuracy degradation, we further propose Budget-Aware (BA) checkpoint selection, which chooses the best checkpoint on the training trajectory subject to a target budget $\epsilon _\rm{max}$, without re-training. For privacy evaluation, we adopt a canary-based suite and emphasize canary negative log-likelihood (NLL) as a more sensitive memorization signal than hit-rate. Experiments on public medical QA benchmarks derived from MedQA and MedMCQA, together with synthetic canary auditing, characterize a clear privacy-utility frontier in a controlled evaluation setting. Non-private LoRA improves average accuracy but increases memorization signals, whereas privacy-budgeted training mitigates memorization while maintaining stable performance under explicit privacy accounting.