Hypothesis testing for neural cell growth experiments using a hybrid branching process model

Neuron branching patterns can characterize neural cell types and act as markers for neurodegenerative disease and neural development. We develop a hybrid Markovian model for neural branching that extends previously published models by (i) using a discretized gamma model to account for underdispersion in primary branching, (ii) incorporating both bifurcation and trifurcation branching events to accommodate observed data, and (iii) only requiring branch counts and not branching topology as observations, allowing larger numbers of neurons to be sampled than in previous literature. Inference for primary branching is achieved through a gamma generalized linear model. Due to incomplete data, bifurcation and trifurcation probabilities are estimated using an expectation-maximization algorithm, which is shown to give consistent estimates using simulation studies and theoretical arguments. In simulation studies, comparison of standard errors shows no significant loss of accuracy relative to when topological information is available. A unified methodology for testing hypotheses using likelihood ratio tests (LRTs) is developed. The methodology is applied to an experiment where neurons are cocultured with different treatments: growth factor (GF), hypothalamic-astroglial conditioned medium (HY), and combination. The model provides statistically adequate fit at all branching orders. All treatments cause significantly higher branching at primary and secondary orders relative to control (p-value < 0.01), but not at higher branching orders, suggesting genetic regulation by the treatments. Using a computationally feasible lower bound on the LRT, bifurcation probabilities are shown to decrease exponentially with branching order for all treatments except HY (p-value 0.03).