BenchMetrics Prob: benchmarking of probabilistic error/loss performance evaluation instruments for binary...

Area of Science:

• Machine Learning
• Statistical Modeling
• Computer Science

Background:

• Probabilistic error/loss metrics, common in regression, are increasingly used for binary classification.
• Existing methods lack systematic evaluation for their suitability in classification tasks.

Purpose of the Study:

• To systematically assess probabilistic instruments for binary classification performance evaluation.
• To identify weaknesses of current metrics and determine the most robust options.

Main Methods:

• A two-stage benchmarking method, BenchMetrics Prob, was developed.
• The method utilized five criteria and fourteen simulation cases with synthetic datasets.
• 31 instrument/instrument variants were tested.

Main Results:

• Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) were identified as the most robust metrics.
• MAE is recommended for general purposes due to its interpretability and [0, 1] range.
• RMSE is preferred when emphasizing larger errors.

Conclusions:

• Researchers should carefully select robust probabilistic metrics for binary classification performance evaluation.
• Metrics like LogLoss, MAPE, sMAPE, and MRAE demonstrated lower robustness and should be avoided.