Which R² should be used to benchmark polygenic score performance against heritability?

The narrow-sense heritability of a trait refers to the proportion of phenotypic variance attributable to additive genetic effects and provides a predictive upper bound for linear polygenic scores. Performance benchmarking in the context of continuous traits is generally performed using the coefficient of determination (R²). Several definitions of R² exist, however, and here we demonstrate that the correlation-based R² is the appropriate formulation for the specific task of heritability benchmarking. For non-linear and machine-learning-based polygenic scores, the more relevant performance ceiling is the broad-sense heritability. However, the uncritical use of R² for the assessment of non-linear models has long been cautioned against. We show that correlation-based R² can still be meaningfully interpreted as the proportion of phenotypic variance recoverable from a model's predictions, even when generated through non-linear means. We therefore highlight the importance of reporting correlation-based R² when benchmarking both traditional and machine-learning-based genomic prediction models against trait heritability.