The STONE Curve: A ROC-Derived Model Performance Assessment Tool

Area of Science:

• Space Physics
• Geophysics
• Numerical Modeling

Background:

• Traditional model validation often relies on categorical classifications.
• Existing methods like the relative operating characteristic (ROC) curve are limited when dealing with continuous data.
• A need exists for advanced tools to assess the performance of numerical models that aim to precisely reproduce observations.

Purpose of the Study:

• Introduce a novel tool, the sliding threshold of observation for numeric evaluation (STONE) curve, for enhanced model validation.
• Adapt the ROC curve technique to leverage the continuous nature of observational data.
• Develop a method applicable to any field using numerical models to reproduce data.

Main Methods:

• The STONE curve utilizes continuous observational data, unlike ROC curves that sort categorical data.
• Thresholds are simultaneously adjusted for both observational and model values when they share the same units and scale.
• The STONE curve plots probability of detection against probability of false detection, similar to ROC curves.

Main Results:

• The STONE curve can be nonmonotonic, exhibiting ripples that indicate data-model value pair asymmetries.
• This technique was successfully applied to modeling geomagnetic activity indices and energetic electron fluxes.
• The STONE curve provides a more nuanced assessment of model performance with continuous data.

Conclusions:

• The STONE curve offers a powerful new approach for validating numerical models across various scientific and engineering disciplines.
• Its ability to detect asymmetries provides deeper insights into model-data relationships.
• This tool advances the field of model performance assessment for continuous datasets.