Responder cell frequency estimation and binomial three-level nonlinear mixed effects model in limiting dilution...

Responder cell frequencies (RCF), which describe vaccine-boosted immune responses in herpes zoster (HZ) prevention studies, have been estimated by using limiting dilution assays (LDA). The theoretical linearity assumption between the logarithm of the proportion of nonresponding wells (s) and the cell concentration (N) (or dilution level) in LDA, based on the single-hit Poisson model, is often violated with observed data resulting in biased estimates of RCF. In this article, the Poisson assumption is modified by applying a mixture of Poisson and gamma distributions, resulting in a negative binomial assumption, which presents a better fit between s and N. In LDA for HZ prevention studies, binary responses (responder or non-responder wells) are measured repeatedly at different cell concentrations and over time. To account for the correlation between responses to varying dilution levels from individuals, and the correlation between repeated assays of individuals over time simultaneously, a binomial three-level nonlinear mixed-effects model is proposed. For parameter estimation, a maximum likelihood method is applied via adaptive Gaussian quadrature. There is a lack of non-Gaussian multilevel nonlinear mixed-effects software, which can execute such a complicated fit. In this article, an algorithm for the three-level nonlinear mixed-effects model, which can be inserted into the code in the SAS procedure NLMIXED, is suggested.