Commonly used software tools produce conflicting and overly-optimistic AUPRC values

Area of Science:

• Bioinformatics
• Machine Learning
• Computational Biology

Background:

• The precision-recall curve (PRC) and area under it (AUPRC) are crucial metrics for evaluating classification models, especially with imbalanced datasets common in fields like cancer diagnosis and cell type annotation.
• These metrics are widely adopted in scientific literature, with over 3,000 studies utilizing tools for PRC plotting and AUPRC computation.

Approach:

• This study systematically evaluated ten popular software tools used for generating PRCs and calculating AUPRC.
• The evaluation focused on the consistency and accuracy of AUPRC values computed by these tools.

Key Points:

• Significant variability was observed in AUPRC values across the evaluated tools.
• The ranking of classification models differed depending on the tool used for AUPRC computation.
• Several tools were found to produce overly optimistic AUPRC estimates, potentially misrepresenting model performance.

Conclusions:

• The choice of tool for AUPRC calculation can impact the assessment of classification performance.
• Researchers should exercise caution when interpreting AUPRC values, considering the potential for tool-specific biases.
• Further standardization or clear guidelines for AUPRC computation are needed to ensure reliable performance evaluation in scientific studies.