Pair-based covariance under mixed mating and its consequences for REML variance estimation

This organism-independent theoretical study examines genomic evaluation of open-pollinated progeny from partially selfing species when the fitted genetic covariance contains only an additive genomic relationship matrix. Open-pollinated families contain selfed-selfed, selfed-outcrossed, and outcrossed-outcrossed sibling pairs, so their covariance must be averaged using joint pair probabilities rather than probabilities assigned to individual offspring. An offspring-level shortcut overestimates expected additive covariance whenever selfing and outcrossing coexist. Extending the pair calculation to the covariance between additive effects and dominance deviations in homozygotes gives a balanced-reference projection coefficient equal to twice the selfing rate under mating-system equilibrium and random unrelated outcrossing. When the solution is interior, this coefficient represents the exact restricted maximum-likelihood projection for balanced families with exchangeable covariance blocks. An independent Mendelian simulation with biallelic quantitative trait loci recovered the classical pair-specific coefficients without supplying them to the data-generating model and closely reproduced the theoretical coefficient, with a coefficient of determination of 0.9998 and mean absolute error of 0.0073. The additive genomic relationship matrix was constructed in a separate controlled marker simulation by VanRaden scaling, and additive-only restricted maximum-likelihood estimates closely followed the balanced-reference prediction, with a coefficient of determination of 0.9900 and mean absolute departure of 0.0107. The resulting dimensionless index and scale conversion are therefore sensitivity diagnostics rather than universal corrections for genomic evaluation designs.