A comprehensive analysis comparing linear and generalized linear models in detecting adaptive SNPs

To understand how organisms adapt to their environment, a gene-environmental association (GEA) analysis is commonly conducted. GEA methods based on mixed models, such as linear latent factor mixed models (LFMM) and LFMM2, have grown in popularity for their robust performance in terms of power and computational speed. However, it is unclear how the assumption of a Gaussian distribution for the response variables influences model performance. In this paper, we develop a generalized linear model (GLM) that allows for non-Gaussian distribution in the genotypic response variables, and treatment of multiallelic nucleotide polymorphisms. Moreover, this multinomial logistic regression model (MLR) is combined with an admixture-based model or principal components analysis to correct for population structure (MLR-ADM and MLR-PC). Using simulations, we evaluate the type 1 error, false discovery rates (FDR), and power to detect selected SNPs, to guide model choice and best practices. With genomic control, MLR-PC and LFMM2 have similar type 1 error, FDRs, and power when analysing biallelic SNPs, while dramatically outperforming models not accounting for population structure. Differences in performance occur under continuous population structure where MLR-PC outperforms LFMM/LFMM2, especially when a larger number of clusters or triallelic SNPs are analysed. The Human Genome Diversity Project (HGDP) data set shows that both MLR-PC and LFMM2 control the inflation of $$ -values. Analysis of the 1,000 Genome Project Phase 3 data set illustrates that MLR-PC and LFMM2 produce consistent results for most significant SNPs, while MLR-PC discovered additional SNPs corresponding to certain genes, suggesting MLR-PC may be a useful alternative to GEA inference.