Reconstructing complex admixture history using a hierarchical model.

Various methods have been proposed to reconstruct admixture histories by analyzing the length of ancestral chromosomal tracts, such as estimating the admixture time and number of admixture events. However, available methods do not explicitly consider the complex admixture structure, which characterizes the joining and mixing patterns of different ancestral populations during the admixture process, and instead assume a simplified one-by-one sequential admixture model. In this study, we proposed a novel approach that considers the non-sequential admixture structure to reconstruct admixture histories. Specifically, we introduced a hierarchical admixture model that incorporated four ancestral populations and developed a new method, called HierarchyMix, which uses the length of ancestral tracts and the number of ancestry switches along genomes to reconstruct the four-way admixture history. By automatically selecting the optimal admixture model using the Bayesian information criterion principles, HierarchyMix effectively estimates the corresponding admixture parameters. Simulation studies confirmed the effectiveness and robustness of HierarchyMix. We also applied HierarchyMix to Uyghurs and Kazakhs, enabling us to reconstruct the admixture histories of Central Asians. Our results highlight the importance of considering complex admixture structures and demonstrate that HierarchyMix is a useful tool for analyzing complex admixture events.