Noise-induced bistability in the fate of cancer phenotypic quasispecies: a bit-strings approach

Tumor cell populations are highly heterogeneous. Such heterogeneity, both at genotypic and phenotypic levels, is a key feature during tumorigenesis. How to investigate the impact of this heterogeneity in the dynamics of tumors cells becomes an important issue. Here we explore a stochastic model describing the competition dynamics between a pool of heterogeneous cancer cells with distinct phenotypes and healthy cells. This model is used to explore the role of demographic fluctuations on the transitions involving tumor clearance. Our results show that for large population sizes, when demographic fluctuations are negligible, there exists a sharp transition responsible for tumor cells extinction at increasing tumor cells' mutation rates. This result is consistent with a mean field model developed for the same system. The mean field model reveals only monostability scenarios, in which either the dominance of the tumor cells or the dominance of the healthy cells is found. Interestingly, the stochastic model shows that for small population sizes the monostability behavior disappears, involving the presence of noise-induced bistability. The impact of the initial populations of cells in the fate of the cell populations is investigated, as well as the transient times towards the healthy and the cancer states.