Issues in performance evaluation for host-pathogen protein interaction prediction.

Area of Science:

• Computational biology
• Infectious disease research
• Machine learning applications

Background:

• Understanding host-pathogen protein-protein interactions (PPIs) is crucial for infectious disease mechanisms and therapeutics.
• Computational predictions, particularly machine learning, aid in identifying promising PPIs.
• Accurate assessment of host-pathogen interaction (HPI) predictors is critical.

Purpose of the Study:

• To evaluate the effectiveness of K-fold cross-validation for HPI prediction generalization.
• To propose and validate more suitable evaluation schemes for HPI prediction.
• To introduce intuitive metrics for HPI predictor performance assessment.

Main Methods:

• Comparison of K-fold cross-validation with leave-one-pathogen-out (LOPO) cross-validation.
• Analysis of HPI prediction performance in scenarios with unknown interacting partners.
• Development and proposal of simpler, interpretable performance metrics.

Main Results:

• K-fold cross-validation inadequately models HPI prediction for proteins with no known interactions.
• Leave-one-pathogen-out (LOPO) cross-validation provides a more realistic performance estimate.
• Existing metrics (AUC-PR, AUC-ROC) are less intuitive for biologists compared to proposed alternatives.

Conclusions:

• LOPO cross-validation is more effective for evaluating HPI predictors in real-world scenarios.
• The choice of cross-validation strategy significantly impacts the perceived accuracy of HPI predictors.
• Simpler, interpretable metrics are needed for effective communication of HPI prediction performance to biologists.