Evaluating classification performance: Receiver operating characteristic and expected utility

One primary advantage of receiver operating characteristic (ROC) analysis is considered to be its ability to quantify classification performance independently of factors such as prior probabilities and utilities of classification outcomes. This article argues the opposite. When evaluating classification performance, ROC analysis should consider prior probabilities and utilities. By developing expected utility lines (EU lines), this article shows the connection between a classifier's ROC curve and expected utility of classification. In particular, EU lines can be used to estimate expected utilities when classifiers operate at any ROC point for any given prior probabilities and utilities. EU lines are useful across all situations-no matter if one examines a single classifier or compares multiple classifiers, if one compares classifiers' potential to maximize expected utilities or classifiers' actual expected utilities, and if the ROC curves are full or partial, continuous or discrete. The connection between ROC and expected utility analyses reveals the common objective underlying these two methods: to maximize expected utility of classification. Particularly, ROC analysis is useful in choosing an optimal classifier and its optimal operating point to maximize expected utility. Yet, choosing a classifier and its operating point (i.e., changing conditional probabilities) is not the only way to increase expected utility. Inspired by parameters involved in estimating expected utility, this article also discusses other approaches to increase expected utility beyond ROC analysis. (PsycInfo Database Record (c) 2024 APA, all rights reserved).