A Coalescent Sampler Successfully Detects Biologically Meaningful Population Structure Overlooked By F-Statistics

Assessing the geographic structure of populations has relied heavily on Sewell Wright's F-statistics and their numerous analogues for many decades. However, it is well appreciated that, due to their nonlinear relationship with gene flow, F-statistics frequently fail to reject the null model of panmixia in species with relatively high levels of gene flow and large population sizes. Coalescent genealogy samplers instead allow a model-selection approach to the characterization of population structure, thereby providing the opportunity for stronger inference. Here, we validate the use of coalescent samplers in a high gene flow context using simulations of a stepping-stone model. In an example case study, we then re-analyze genetic datasets from 41 marine species sampled from throughout the Hawaiian archipelago using coalescent model selection. Due to the archipelago's linear nature, it is expected that most species will conform to some sort of stepping-stone model (leading to an expected pattern of isolation by distance), but F-statistics have only supported this inference in similar to 10% of these datasets. Our simulation analysis shows that a coalescent sampler can make a correct inference of stepping-stone gene flow in nearly 100% of cases where gene flow is <= 100 migrants per generation (equivalent to F-ST = 0.002), while F-statistics had mixed results. Our re-analysis of empirical datasets found that nearly 70% of datasets with an unambiguous result fit a stepping-stone model with varying population sizes and rates of gene flow, although 37% of datasets yielded ambiguous results. Together, our results demonstrate that coalescent samplers hold great promise for detecting weak but meaningful population structure, and defining appropriate management units.