Beware of External Validation! - A Comparative Study of Several Validation Techniques used in QSAR Modelling

Area of Science:

• Quantitative Structure-Activity Relationship (QSAR) modeling
• Cheminformatics
• Statistical modeling

Background:

• QSAR model validation is crucial for reliable predictions.
• External validation is common but its efficacy is questionable for high-dimensional, small-sample datasets (n << p).
• Advancements in computation enable calculation of numerous molecular descriptors, leading to datasets with more predictors than samples.

Purpose of the Study:

• To extensively compare external validation with other methods for QSAR models.
• To evaluate validation techniques on datasets with a high number of predictors and a small number of samples.
• To identify the most reliable validation method for such challenging datasets.

Main Methods:

• Compared Leave-one-out (LOO), K-fold, external, and multi-split validation.
• Utilized LASSO regression for simultaneous variable selection and modeling.
• Evaluated methods using 300 simulated datasets and one real dataset of 95 amine mutagens.

Main Results:

• External validation metrics exhibited high variability across data splits, indicating instability.
• Leave-one-out (LOO) validation showed the best overall performance in the evaluated scenarios.
• External validation is not recommended for predictive QSAR models with high-dimensional, small-sample data.

Conclusions:

• External validation results are too unstable for high-dimensional, small-sample QSAR datasets.
• The Leave-one-out (LOO) procedure is recommended for validating QSAR predictive models in these data-rich, sample-poor conditions.