BAYESIAN SELECTION AND REFITTING OF REFERENCE MUTATIONAL SIGNATURES IN CANCER GENOMICS

Somatic mutations, which accumulate in cells after conception, drive cancer development. Their characteristic patterns, namely mutational signatures, reflect the underlying mutational processes and have provided valuable insights into cancer etiology, evolution and therapeutic strategies. While non-negative matrix factorization (NMF) is commonly used to infer de novo mutational signatures and their activities, it requires large datasets for reliable estimation. When the sample size is limited, signature refitting is typically used, which estimates the signature activities using a set of reference signatures derived from external studies. However, current signature refitting methods often use the full list of reference signatures, leading to overfitting and compromised interpretability and accuracy. Despite its importance, the problem of selecting an appropriate subset of reference signatures received little attention. We proposed BayesSigRefitting, a Bayesian model selection framework to select an optimal subset of reference signatures for accurate refitting. Our approach employs a Bayesian hierarchical model with a sparsity-inducing Laplace prior, and the Shotgun Stochastic Search (SSS) algorithm to efficiently explore possible signature subsets and identify the optimal one. We established the model selection consistency of BayesSigRefitting and demonstrated, through simulation and real data studies across seven cancer types, that it outperformed existing methods in both signature selection and signature activity estimation. These findings highlight the potential of BayesSigRefitting to enhance the accuracy and reliability of mutational signature analysis, especially in settings with limited sample sizes.