The Costs of Indeterminacy: How to Determine Them?

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Indeterminate classifiers provide cautious predictions by assigning multiple classes when uncertainty is high.
• Evaluating indeterminate classifiers is challenging, particularly when prediction errors have unequal costs.

Purpose of the Study:

• To address the performance evaluation of indeterminate classifiers under generic cost-sensitive settings.
• To propose a generalized evaluation framework beyond the standard accuracy (0/1 costs) setting.

Main Methods:

• Defining desirable properties for the costs associated with indeterminate predictions.
• Reviewing existing evaluation methods against these properties.
• Developing a novel, general formula for producing and evaluating indeterminate predictions.

Main Results:

• The proposed formula aligns with the identified desirable properties for cost-sensitive evaluation.
• Experiments demonstrate the effectiveness of the new criterion in ordinal regression tasks.
• The study provides a unified approach for handling indeterminate predictions across different cost scenarios.

Conclusions:

• A robust method for evaluating indeterminate classifiers with generic cost functions has been developed.
• The proposed approach enhances the practical application of cautious AI models in real-world scenarios.
• This work offers a significant advancement in the field of cost-sensitive machine learning evaluation.