Mind your prevalence!

Area of Science:

• Computational chemistry and cheminformatics
• Machine learning model validation
• Quantitative structure-activity relationship (QSAR) studies

Background:

• Standard performance metrics like accuracy can be misleading in binary classification due to imbalanced datasets.
• Balanced accuracy and Matthews' correlation coefficient (MCC) are used to mitigate imbalance but have limitations.
• Existing literature lacks a formal framework for understanding and extending balanced metrics beyond balanced accuracy.

Purpose of the Study:

• To formally define confusion matrix, sensitivity, and specificity in the context of QSAR model validation.
• To demonstrate the limitations of comparing performance metrics under non-constant prevalence.
• To extend the concept of balanced metrics to Matthews' correlation coefficient (MCC) and other performance indicators.

Main Methods:

• Formal definition of confusion matrix, sensitivity, and specificity.
• Analysis of prevalence dependence using synthetic data.
• Derivation of balanced metric expressions as functions of sensitivity, specificity, and prevalence.

Main Results:

• Balanced accuracy is equivalent to accuracy calibrated to a 50% prevalence (balanced test set).
• Matthews' correlation coefficient (MCC) is shown to be prevalence-dependent, with potential underestimation at extreme prevalences.
• A novel framework is presented for balanced metrics, including a balanced MCC, expressed via sensitivity and specificity.

Conclusions:

• The proposed balanced metrics offer a robust and unified approach for QSAR model validation, enabling reliable comparisons across datasets with varying prevalences.
• This work extends the concept of balanced evaluation beyond balanced accuracy, providing a more comprehensive understanding of model performance.
• The derived confusion matrix and balanced metric expressions facilitate accurate interpretation and comparison of QSAR models in cheminformatics.