Using simulations to evaluate Mantel-based methods for assessing landscape resistance to gene flow

Area of Science:

• Landscape genetics
• Spatial ecology
• Population genetics

Background:

• Mantel-based tests are standard for analyzing spatial genetic structure influenced by landscape features.
• Previous simulation studies revealed challenges and sparked debate regarding the appropriate use of Mantel tests in landscape genetics.

Purpose of the Study:

• To clarify the debate surrounding Mantel tests by examining their performance using spatially explicit, individual-based genetic simulations.
• To investigate the impact of landscape configuration, spatial genetic nonequilibrium, nonlinear relationships, and resistance model correlations on Mantel test efficacy.

Main Methods:

• Spatially explicit, individual-based genetic simulations were employed.
• The study examined the performance of Mantel-based methods under varying conditions, including landscape configuration and genetic distance relationships.
• Causal modeling and relative support metrics were used to assess model identification accuracy.

Main Results:

• Mantel-based methods generally performed poorly across most simulated conditions.
• Performance improved significantly when landscapes were fragmented, spatial genetic equilibrium was achieved, and relationships between genetic and cost distances were linearized.
• Model identification accuracy increased with relative support and simple Mantel r values, reaching approximately 50%.

Conclusions:

• Mantel tests with linearized relationships are suitable for differentiating resistance models with specific cost distance correlations.
• The effectiveness of Mantel tests for fine-tuning resistance values is limited due to high correlations among resistance model outputs.