Rank order entropy: why one metric is not enough

Area of Science:

• Computational Chemistry
• Cheminformatics
• Drug Discovery

Background:

• Quantitative Structure-Activity Relationship (QSAR) models are widely used in drug discovery for property prediction.
• Traditional QSAR validation metrics (e.g., r², PRESS r²) assess internal consistency but not prediction stability against training set changes.
• Misapplication of poorly constructed QSAR models or those used outside their applicability domain has led to historical challenges.

Purpose of the Study:

• To develop and evaluate a novel metric for quantifying the stability of QSAR rank order predictions.
• To assess QSAR model performance and reliability by measuring their response to systematic reductions in training data.
• To provide a method for setting realistic user expectations regarding QSAR model predictive performance.

Main Methods:

• Examined QSAR rank order model stabilities using representative datasets, descriptor sets, and modeling methods.
• Employed Kendall Tau as a rank order metric and Shannon entropy to quantify rank-order stability.
• Utilized Data Truncation Analysis (DTA) to systematically reduce training set information content and evaluate model response.

Main Results:

• The novel rank order entropy (ROE) metric was applied to 71 datasets, revealing more about model behavior than traditional metrics alone.
• Found that stable models did not necessarily predict rank order well, and well-performing rank order models did not always excel in traditional metrics.
• ROE evaluation identified specific QSAR models and combinations of data/descriptor sets and modeling methods that should be discarded for prioritization schemes.

Conclusions:

• ROE provides a crucial measure of QSAR model stability, complementing traditional validation metrics.
• This metric helps discern usable QSAR models for specific applications and enhances confidence in their predictions within defined domains of applicability.
• The study highlights the importance of evaluating model stability alongside predictive accuracy for reliable QSAR applications in drug discovery.