Genome scan methods against more complex models: when and how much should we trust them?

Area of Science:

• Population Genetics
• Genomics
• Evolutionary Biology

Background:

• Next-generation sequencing (NGS) provides dense genetic markers for detecting selection.
• Genome scan methods are crucial for identifying genomic regions under selection.
• Previous studies often overlooked complex population structures and polygenic selection.

Purpose of the Study:

• To investigate the power and error rates of popular genome scan methods.
• To evaluate methods under complex, hierarchical population structures and polygenic selection.
• To compare frequentist and Bayesian approaches using a false discovery rate (FDR) framework.

Main Methods:

• Individual-based simulation study.
• Evaluation of linear regression, Bayescan, BayEnv, and LFMM.
• Assessment of population allele frequencies versus individual genotype data for LFMM and linear regression.

Main Results:

• Method performance ranking is influenced by polygenic selection.
• Power can be very low in strongly hierarchical scenarios with confounding demographic and environmental effects.
• LFMM demonstrated the best power-error rate compromise across scenarios; combining methods reduced error rates.

Conclusions:

• LFMM is a robust method for detecting selection under complex scenarios.
• Considering results from multiple methods enhances the reliability of identifying outlier loci.
• Complex population structure and polygenic selection significantly impact the performance of genome scan methods.