Mutation, multilevel selection, and the evolution of propagule size during the origin of multicellularity

Evolutionary transitions require the organization of genetic variation at two (or more) levels of selection so that fitness heritability may emerge at the new level. In this article, we consider the consequences for fitness variation and heritability of two of the main modes of reproduction used in multicellular organisms: vegetative reproduction and single-cell reproduction. We study a model where simple cell colonies reproduce by fragments or propagules of differing size, with mutations occurring during colony growth. Mutations are deleterious at the colony level but can be advantageous or deleterious at the cell level ("selfish" or "uniformly deleterious" mutants). Fragment size affects fitness in two ways: through a direct effect on adult group size (which in turn affects fitness) and by affecting the within- and between-group variances and opportunity for selection on mutations at the two levels. We show that the evolution of fragment size is determined primarily by its direct effects on group size except when mutations are selfish. When mutations are selfish, smaller propagule size may be selected, including single-cell reproduction, even though smaller propagule size has a direct fitness cost by virtue of producing smaller organisms, that is, smaller adult cell groups.