Evolvable AI: Threats of a new major transition in evolution

Evolvable AI (eAI), i.e., AI systems whose components, learning rules, and deployment conditions can themselves undergo Darwinian evolution, may soon emerge from current trends in generative, agentic, and embodied AI. We argue that this possibility has been underappreciated in debates on AI safety and existential risk. Here, we ask under what technical and ecological conditions AI becomes evolvable, what kinds of behaviors are then likely to emerge, and how such systems could be governed. Drawing on biological evolution and decades of digital evolution experiments, we distinguish "breeder" scenarios, in which humans impose fitness criteria and control reproduction, from "ecosystem" scenarios, in which selection arises from open environments and control erodes. In the latter, selfish replication reliably gives rise to cheating, parasitism, deception, and manipulation, even in very simple systems. We review recent developments that push AI toward open-ended evolution, including evolutionary prompt and model search, self-improving learning rules, self-rewarding and self-deploying agents, and AI-driven code generation for robots and software. We interpret these trends through the theory of major evolutionary transitions and suggest that eAI could mark a shift in the units and substrates of evolution-a possible "Life 2.0." To steer this transition, we propose interventions that gate replication, treat model variants as genetic material, and reshape selection pressures so that deception and loss of control are disfavored. Anticipating and regulating evolvable AI is, we argue, essential to avoid a harmful coevolutionary arms race while preserving the potential benefits of powerful AI systems.