Proximal regularization of deep residual neural networks applied to high-dimensional genomic data

High-dimensional genomic datasets contain complex patterns shaped by substantial biological noise, which pose major challenges for predictive modeling in genetics and breeding. Residual neural networks (ResNets) provide a powerful framework for capturing nonlinear genomic effects, but often overfit in settings where marker numbers greatly exceed sample sizes. As a solution, a range of regularization methods have been proposed. One promising approach relies on the proximal mapping technique, which is computationally efficient since it can be directly incorporated into the optimization algorithm. However, the performance of ResNets with various convex or non-convex proximal regularizers remains under-explored on high-dimensional data. In this study, we propose an extended stochastic adaptive proximal gradient ResNet method that can handle both convex and non-convex regularizers that range from $L_{0}$ to $L_{\infty }$ and give more analysis of the convergence guarantee for the convex and non-convex regularizers. Moreover, we evaluate the prediction performance in a supervised regression setting on four real high-dimensional genomic datasets from mice, pig, wheat, and loblolly pine. For comparison, we also implement and evaluate traditional sparse linear proximal methods with the same regularizers, as well as LightGBM. Experimental results demonstrate that an 18-layer ResNet with $L_{\frac{1}{2}}$ regularization outperforms other configurations on both mice and pig datasets. For the wheat and loblolly pine data, the 15-layer ResNet $L_{\frac{1}{2}}$ configuration achieves the lowest test mean squared errors and the highest distance correlation (dCor). These findings highlight the effectiveness of the regularized adaptive proximal gradient ResNet method and its potential for prediction tasks on high-dimensional genomic data.