Evaluation of AI-Based Medical Device Concerning Localization Information Using Nonparametric Inference for the...

The alternative free-response receiver operating characteristic (AFROC) curve is a popular method for evaluating the performance of diagnostic tests concerning detecting and locating abnormal lesions. However, the existing inferences for the AFROC curve rely on the assumptions of the independence of the observations within the same subject and certain parametric models, which are hard to test and may not be true in practice. In this article, we propose nonparametric inferences for the AFROC curve. Under reasonable assumptions, we derive the asymptotic properties for the empirical AFROC curve and use them to make inferences about the AFROC curve and its related indices. We propose a new bootstrap method to construct the confidence intervals of the indices related to the AFROC curve and the confidence band for the AFROC curve. Simulations show that our method significantly outperforms the existing parametric approach when its assumptions are violated. We also provide a real-world example, that is, a diagnostic test investigating AI-assisted pulmonary nodule diagnosis, to illustrate the practical applicability of our method.