Performance of Regression Models as a Function of Experiment Noise

Area of Science:

• Machine Learning
• Computational Biology
• Statistical Modeling

Background:

• Developing machine learning regression models faces challenges in determining if peak performance on test datasets has been achieved.
• Biological data often contains experimental noise in response variables, limiting attainable model performance metrics.
• This inherent label noise poses a fundamental constraint on regression model evaluation.

Purpose of the Study:

• To derive an expected upper bound for the coefficient of determination (R ²) for regression models tested on holdout datasets.
• To provide a method for assessing the maximum potential performance of regression models, especially in the presence of response variable noise.
• To aid researchers in optimizing machine learning regression models by establishing performance benchmarks.

Main Methods:

• Derived an expected upper bound for R ² based on response variable noise and variance.
• Validated the upper bound estimate using Monte Carlo simulations.
• Applied the upper bound estimation to bootstrap performance of regression models on biological datasets (protein sequence, transcriptomic, genomic data).

Main Results:

• An expected upper bound for R ² was successfully derived, dependent only on response variable noise and variance.
• The derived upper bound was validated through simulations, confirming its reliability.
• The method was demonstrated on diverse biological datasets, showing its practical utility in assessing model performance.

Conclusions:

• The novel method for estimating upper bounds of test data performance aids researchers in developing optimal ML regression models.
• The derived upper bounds are applicable to regression models across various fields, not limited to biological data.
• This approach helps researchers ascertain if their machine learning models have reached their maximum potential performance.