Comparing the Influence of Simulated Experimental Errors on 12 Machine Learning Algorithms in Bioactivity Modeling...

Area of Science:

• Computational chemistry
• Cheminformatics
• Machine learning

Background:

• Quantitative Structure-Activity Relationship (QSAR) models are crucial in drug discovery.
• The impact of random experimental errors (noise) on QSAR model predictive power is not well understood.
• Systematic studies are needed to assess algorithm robustness against noise.

Purpose of the Study:

• To systematically evaluate the noise sensitivity of 12 diverse machine learning algorithms used in QSAR modeling.
• To provide a practical guide for selecting appropriate algorithms and parameters based on data noise levels.
• To benchmark algorithm performance under varying degrees of simulated experimental noise.

Main Methods:

• 15,840 QSAR models were generated using 12 algorithms (SVM, GP, RVM, RF, GBM, Bagged Trees, PLS, kNN) across 12 datasets.
• Noise was simulated by adding Gaussian-distributed random errors to training data bioactivities, with variances up to the data range.
• Model performance on a test set was monitored to assess noise sensitivity.
• A full-factorial experimental design and linear model analysis identified general trends.

Main Results:

• Gradient Boosting Machines (GBM) exhibited the lowest noise tolerance, though performance was similar to RF, SVM (radial/poly), and GP (radial/poly) at low noise levels.
• The 'bag fraction' parameter significantly impacts GBM's noise sensitivity; low values (0.1-0.2) are recommended for noisy data.
• Most other algorithms (SVM, GP, RF, RVM, PLS, kNN) showed comparable, linear performance degradation with increasing noise.
• Support Vector Machines with polynomial kernels (SVM Poly) and Gaussian Processes with polynomial kernels (GP Poly) demonstrated notable noise sensitivity at high noise levels.

Conclusions:

• Algorithm choice and parameter tuning are critical for maintaining QSAR model predictive power in the presence of experimental noise.
• GBM requires careful parameter selection (e.g., low bag fraction) when dealing with noisy bioactivity data.
• The study offers valuable insights for cheminformaticians and computational chemists to select robust QSAR models for real-world applications.