Variational inference in coupled models of amino acid substitution

We investigate the use of Expectation-Maximization (em) and variational Bayes for inferring rates and interactions under models of molecular coevolution. We first review em theory for continuous-time Markov chains (ctmcs) and develop it for coevolutionary models, exploiting exchangeability and reversibility symmetries to constrain the parameter dimension. We fit several paired amino-acid coevolutionary models to structural alignments and compare the results to previous work. Our richest model trained on pooled coevolutionary data has explanatory power comparable to CherryML's Q2 matrix (also trained on pooled data), with one quarter the parameters. However, we observe that aggregation of training data can lead to a form of Simpson's Paradox: a mixture model, whose components are parameter-efficient continuous-time Bayes networks (ctbns), resolves signals that wash out when a single model tries to capture everything. These signals include both correlated and anticorrelated hydropathy and volume-packing in coevolving amino-acid pairs, as well as the anticorrelated acid/base compensation that was detected by CherryML's Q2. We next present a closed-form evidence lower bound (ELBO) for ctbns using em statistics. Compared to the state of the art in variational modeling of ctbns, the Euler-Lagrange equations derived by Cohn et al (JMLR, 2010), our closed-form ELBO is competitive in accuracy, considerably more efficient, simpler, and more stable. We conclude by describing a covariant indel model: a Dirichlet process selecting coevolving sites within TKF92, yielding an Infinite Pair HMM over alignments and structures. This model slightly outperforms TKF92 on a structural-alignment benchmark. Code and data are at https://tkfdp.net/.