Overrepresentation Bias Leads to Performance Overestimation in Blood-Brain Barrier Permeability Prediction Models:...

Area of Science:

• Computational chemistry
• Pharmacology
• Machine learning in drug discovery

Background:

• Machine learning, especially deep learning, is increasingly used for predicting drug properties like blood-brain barrier permeability (BBBP).
• Model evaluation is critical for credibility but often overlooked compared to feature engineering and model design.
• Drug property databases may contain overrepresentation bias, with near-identical compounds and property values.

Purpose of the Study:

• To investigate the impact of overrepresentation bias on BBBP prediction model performance.
• To propose methods for detecting and mitigating this bias in cheminformatics datasets.
• To emphasize the importance of addressing data bias for reliable model evaluation in drug discovery.

Main Methods:

• Analysis of drug property databases for the presence of overrepresentation bias.
• Quantification of performance metric inflation caused by this bias using metrics like AUC and F1-score.
• Development of an automatic algorithm for bias detection and a bias-aware data handling procedure.

Main Results:

• Overrepresentation bias was found to significantly inflate performance estimates in BBBP prediction models.
• Average inflation was 13.3% for Area Under Curve (AUC) and 16.44% for macro F1-score.
• The proposed detection algorithm and data handling procedure offer a solution to mitigate this bias.

Conclusions:

• Overrepresentation bias poses a significant threat to the reliability of machine learning models in drug property prediction.
• Addressing this bias is more impactful than optimizing feature selection or model architecture.
• The study recommends adopting bias-aware methodologies to ensure robust and trustworthy evaluations in drug discovery research.