Be aware of overfitting by hyperparameter optimization!

Area of Science:

• Computational Chemistry
• Machine Learning
• Drug Discovery

Background:

• Hyperparameter optimization is common in machine learning for tasks like solubility prediction.
• Previous studies utilized graph-based methods on diverse solubility datasets.
• Concerns exist regarding potential overfitting during extensive hyperparameter tuning.

Purpose of the Study:

• To investigate the impact of hyperparameter optimization on model performance in solubility prediction.
• To compare the efficiency and accuracy of pre-set hyperparameters versus optimized ones.
• To evaluate a novel Natural Language Processing-based representation learning method, Transformer CNN.

Main Methods:

• Analysis of seven thermodynamic and kinetic solubility datasets.
• Comparison of state-of-the-art graph-based methods with hyperparameter optimization and pre-set hyperparameters.
• Implementation and evaluation of Transformer CNN, a Natural Language Processing approach using SMILES strings.

Main Results:

• Hyperparameter optimization did not consistently improve model performance and could lead to overfitting.
• Models with pre-set hyperparameters achieved comparable results to optimized models, reducing computational cost by approximately 10,000 times.
• Transformer CNN outperformed graph-based methods in 26 out of 28 comparisons, demonstrating superior accuracy and efficiency.

Conclusions:

• Pre-optimized hyperparameters can negatively impact model generalization due to overfitting.
• Utilizing pre-set hyperparameters is a computationally efficient strategy yielding comparable predictive performance.
• Transformer CNN represents a significant advancement in solubility prediction accuracy and speed.