Fingering instability at the interface between growing and not-growing cell populations with different mobilities

Invasion of solid tumours is often mediated by heterogeneous cell populations that differ in motility, adhesion, and proliferation. Motivated by these mechanisms, we study a two-population continuum model in which the total density generates a pressure field and cell transport follows a Darcy-type law, coupled to growth and nonlinear diffusion. The model admits planar travelling-wave solutions describing an advancing invasion front. We derive, in the planar travelling-wave geometry and in the incompressible limit, an explicit dispersion relation for transverse perturbations of segregated fronts. The instability results in the amplification of interfacial modes and the emergence of fingering patterns, and numerical simulations near the incompressible limit recover the growth rates predicted by the linear analysis. To validate these predictions, we design a fully implicit positivity-preserving numerical scheme based on a projected Newton-Krylov solver, which enforces non-negativity of the densities at the algebraic level and remains robust in stiff regimes with sharp gradients, to avoid spurious front breakup. Numerical experiments reproduce the predicted early-time amplification at the most unstable modes considered. Altogether, the results link phenotypic heterogeneity and mechanically mediated transport to morphological instabilities of tumour invasion fronts, with potential relevance to interfaces established by adhesion-mediated cell sorting.