DeepHyb: deep learning-based inference of hybridization with embedded comprehensive multiple sequence alignment features

Hybridization is a pivotal evolutionary process that shapes speciation and genetic diversity across the tree of life. Traditional methods for hybridization inference rely on site-pattern summary statistics and lack the ability to capture higher-order sequence dependencies, limiting their performance in complex scenarios. Additionally, existing tools often fail to explicitly classify non-hybrid evolutionary scenarios or assign parental-hybrid lineage identities, creating challenges for interpreting results in genomic studies. There is a critical need for interpretable, efficient methods that leverage full genomic sequence information to address these limitations.

DeepHyb is a convolutional neural network (CNN)-based deep learning framework designed to infer historical hybridization events directly from multiple sequence alignment-derived site-pattern and k-mer count features. It integrates four complementary feature representations-15 summary site patterns, 256 one-base site patterns, 75 summary k-mer patterns, and 256 sequence-level k-mer patterns-to capture both short-range and higher-order sequence dependencies. In the canonical four-taxon scenario, which serves as the standard framework for evaluating phylogenetic discordance under the multispecies coalescent (one outgroup + three ingroups), DeepHyb jointly predicts hybridization status and parental-hybrid lineage combinations. On simulated data under the setting of sequence length = 100,000 and mutation rate = 0.01, our method performs with high accuracy, achieving an accuracy of 0.8773, precision of 0.5000, and F1 score of 0.6667 (vs. HyDe's 0.8501, 0.4501, and 0.6202), while maintaining a recall of 1.0. On the empirical Heliconius dataset, DeepHyb reproduces the HyDe-derived hybrid/non-hybrid labels with over 93% concordance; no independent ground truth of true hybridization events is available for these. Through Shapley additive explanations (SHAP), DeepHyb identified three highly informative phylogenetic-invariant site-pattern features, ABBA, ABAA, and ABAB, which correspond to the core quartet contrasts used by HyDe-based hybridization inference. Unlike HyDe-CNN, which requires chromosome-scale contiguous alignments, DeepHyb operates on MSA-derived site-pattern and k-mer feature representations, enabling application to fragmented or non-contiguous genomic data while retaining interpretable signals of hybridization/introgression. The framework is implemented in Python, supports multi-core parallel execution and GPU acceleration, and is freely available as an open-source resource.

DeepHyb established an interpretable, proof-of-concept approach demonstrated on simulated data and in Heliconius, whose broader generalizability across taxa remains to be established for hybridization inference by bridging data-driven deep learning with biological insight. It addresses key limitations of traditional tools, offers robust performance in both simulated and empirical datasets, and uncovers novel genomic signatures of hybridization. This tool is valuable for evolutionary genomics research, enabling precise detection of hybridization events, assignment of parental-hybrid relationships, and exploration of introgression dynamics across diverse taxa.