How Precise Are Our Quantitative Structure-Activity Relationship Derived Predictions for New Query Chemicals?

Area of Science:

• Computational Chemistry
• Cheminformatics
• Drug Discovery

Background:

• Quantitative Structure-Activity Relationship (QSAR) models are crucial for prediction and data gap filling across various scientific domains.
• Traditional QSAR model validation relies on test sets with available experimental data.
• Assessing prediction reliability for entirely new datasets, even within the applicability domain, remains a challenge.

Purpose of the Study:

• To develop a robust method for evaluating the reliability of QSAR model predictions on external datasets.
• To categorize prediction quality into 'good', 'moderate', and 'bad' based on prediction errors.
• To create a composite score that reflects prediction reliability using multiple criteria.

Main Methods:

• Categorized prediction quality based on absolute prediction errors.
• Utilized three criteria: leave-one-out error, similarity-based applicability domain (AD), and proximity to mean training response.
• Applied different weighting schemes to generate a composite prediction score.
• Validated the scheme using 5 diverse datasets and 15 models each, across three splitting techniques.

Main Results:

• A weighting scheme of 0.5-0-0.5 demonstrated high concordance (>80%) between composite score-based and error-based categorization.
• The findings were confirmed with true external sets for four additional endpoints.
• The developed scheme shows broad applicability for judging prediction reliability on new datasets.

Conclusions:

• The proposed composite scoring scheme effectively predicts the reliability of QSAR models for external datasets.
• This approach enhances the trustworthiness and utility of QSAR predictions in scientific research.
• A tool, 'Prediction Reliability Indicator', has been developed and is available online for multiple linear regression models.