Robust non-linear mixed modelling of longitudinal PSA levels after prostate cancer treatment

The objective of this study was to develop a robust non-linear mixed model for prostate-specific antigen (PSA) measurements after a high-intensity focused ultrasound (HIFU) treatment for prostate cancer. The characteristics of these data are the presence of outlying values and non-normal random effects. A numerical study proved that parameter estimates can be biased if these characteristics are not taken into account. The intra-patient variability was described by a Student-t distribution and Dirichlet process priors were assumed for non-normal random effects; a process that limited the bias and provided more efficient parameter estimates than a classical mixed model with normal residuals and random effects. It was applied to the determination of the best dynamic PSA criterion for the diagnosis of prostate cancer recurrence, but could be used in studies that rely on PSA data to improve prognosis or compare treatment efficiencies and also with other longitudinal biomarkers that, such as PSA, present outlying values and non-normal random effects.