Dynamic nested hierarchies: self-evolving machine learning architectures for lifelong learning

Contemporary machine learning models, including large language models, exhibit remarkable capabilities in static tasks yet falter in non-stationary environments due to rigid architectures that hinder continual adaptation and lifelong learning. Building upon the nested learning (NL) paradigm, which decomposes models into multi-level optimization problems with fixed update frequencies, this work proposes Dynamic Nested Hierarchies (DNH) as an extension enabling autonomous structural adaptation. Unlike static nested learning where hierarchy depth and update frequencies are fixed at initialization, DNH introduces three biologically-grounded mechanisms: (1) level addition triggered by meta-loss thresholds, analogous to adult neurogenesis in the hippocampal dentate gyrus; (2) level pruning based on gradient contribution, analogous to synaptic elimination; and (3) frequency modulation driven by local surprise signals, analogous to neural oscillation adaptation. We provide explicit mappings between these mechanisms and neuroplasticity processes, moving beyond superficial analogy to principled design. Through rigorous mathematical formulations, we prove convergence bounds of O(1/T + δ²) in non-stationary environments, expressivity improvements bounded by ϵ ≤ O(1/L _t ) + γδ, and sublinear regret $O\left(\sqrt{T}\right)$ compared to static architectures' linear regret. Empirical evaluations on language modeling, continual learning benchmarks-including comparisons with Elastic Weight Consolidation (EWC) and Synaptic Intelligence (SI) as well as modern methods (DER++, MEMO) on Split ImageNet, CLEAR-100, and CORe50-and long-context reasoning validate the theoretical advantages. Comprehensive ablation studies verify the contribution of each component, including the Self-Modifying Memory (SMM) module and Evolutionary Adam (EAdam) optimizer. We provide detailed computational cost analysis and parameter trajectory visualizations demonstrating bounded growth through self-regulating pruning.