Use of a generalized linear mixed model to reduce excessive heterogeneity in petroleum spray oil bioassay data

High heterogeneity (variance) is a consistent and significant problem in petroleum spray oil derived bioassay data. It can mask small statistical differences sought by researchers in relative toxicity or potency analysis. To compensate for excessive heterogeneity, researchers often use very large sample sizes to improve statistical accuracy. We present a statistical method of modeling heterogeneity extending the conventional probit model by adding random effects to it. We illustrate this by reanalyzing 26 of our own published experiments. Twelve of these had excessive heterogeneity that was significantly reduced in ten cases by including random replicate effects with or without random slopes. Five were further improved by allowing a nonlinear (spline) response. The result was tighter confidence intervals for the estimates of lethal dose.