Commonly used software tools produce conflicting and overly-optimistic AUPRC values

Area of Science:

• Bioinformatics
• Machine Learning
• Computational Biology

Background:

• Precision-recall curves (PRC) and area under the PRC (AUPRC) are crucial metrics for evaluating classification performance, especially in imbalanced datasets.
• These metrics are widely applied in fields like cancer diagnosis and cell type annotation.
• Over 3000 published studies have utilized tools for PRC plotting and AUPRC computation.

Purpose of the Study:

• To evaluate the performance and consistency of 10 popular tools used for generating precision-recall curves (PRC) and calculating the area under the PRC (AUPRC).
• To identify discrepancies in classifier ranking and potential biases in AUPRC values reported by different software tools.

Main Methods:

• A comparative analysis of 10 commonly used bioinformatics tools for PRC plotting and AUPRC calculation.
• Assessment of the consistency and accuracy of AUPRC values generated by these tools across various datasets.
• Evaluation of how different tools impact the ranking of classification models.

Main Results:

• Significant variability was observed in the area under the precision-recall curve (AUPRC) values computed by the evaluated tools.
• The ranking of classifiers differed substantially depending on the tool used for AUPRC calculation.
• Several tools were found to produce overly optimistic AUPRC results, potentially misrepresenting classifier performance.

Conclusions:

• The choice of tool for plotting precision-recall curves and computing area under the PRC significantly impacts classification performance evaluation.
• Users should exercise caution as some tools may yield inflated AUPRC values, leading to inaccurate assessments and rankings of classifiers.
• Further standardization or clear guidelines are needed for reliable AUPRC computation in machine learning applications, particularly for imbalanced data.