Analysis of incidence data in developmental toxicity studies: Statistical tests to account for litter effects in...

When analyzing fetal defect incidence in laboratory animal studies, correlation in responses within litters (i.e., litter effects) can lead to increased false-positive rates if litter effects are not incorporated into the analysis. Studies of fetal defects require analysis methods that are robust across a broad range of defect types, including those with zero or near-zero incidence rates in control groups.

A simulation study compared power and false-positive rates for six approaches across a range of background defect rates and litter size distributions. Statistical methods evaluated included ignoring the litter effect as well as parametric and nonparametric approaches based on litter proportions, generalized linear mixed models (GLMMs), the Rao-Scott Cochran-Armitage (RSCA) trend test, and a modification to the RSCA (mRSCA) introduced here to improve estimation at low background rates. These methods were also applied to a common and a rare defect from two prenatal developmental toxicology studies conducted by the National Toxicology Program (NTP).

At background defect rates of 1%, the mRSCA and parametric litter proportion methods provided gains in power over the nonparametric litter proportion method, the GLMM method, and the RSCA method. Simulations involving litter loss in high-dose groups showed loss of power for both litter proportion methods.

The mRSCA test developed here compares favorably with other litter-based approaches and is robust across a range of background defect rates and litter size distributions, making it a practical choice for prenatal developmental toxicology studies involving both common and rare fetal defects.