Clustered binary logistic regression in teratology data using a finite mixture distribution

The beta-binomial distribution introduced by Skellam has been applied in many teratology problems for modelling the litter effect. Recently, Morel and Nagaraj proposed a new distribution for modelling cluster multinomial data when the clustering is believed to be caused by clumped sampling. It turns out that the distribution is a mixture of two binomial distributions and accommodates the estimation of an additional parameter to account for intra-litter effect. The new distribution arises from a cluster mechanism in which some individuals within a cluster exhibit the same behaviour while the remaining individuals from the cluster react independently of each other. Such a mechanism is a natural model in teratology problems, where typically a genetic trait is passed with a certain probability to the foetuses of the same litter. In this article, we use the new distribution to model binary responses with logistic regression. We analyse data from a teratology experiment to demonstrate that the new model provides a useful addition to current methodology. The experiment investigates the synergistic effect of the anticonvulsant phenytoin and trichloropopene oxide on the prenatal development of inbred mice. In a simulation study we investigate the type I error rate and the power of the maximum likelihood ratio test when the data follow a finite mixture distribution.