A minimal net reclassification improvement to assess predictions of intensive care mortality

In assessing the improved discrimination of a new prognostic score, the "Net Reclassification Improvement" from reclassification methods appears of interest. We propose a measure that takes into account improvements in predicted probabilities to assess and allows testing the additional predictive ability of a new scoring system Y compared to a reference score X.

To assess and test the improvement in mortality prediction of (X + Y) compared to X, we defined a minimal net reclassification improvement that restricted improvements in predicted probabilities according to some positive threshold δ. Both absolute and relative improvements were considered. A simulation study was performed to assess its performances in a range of practical situations. We then applied our measures to real intensive care unit data.

Expectedly, minimal net reclassification improvement increased with the effect size of Y and decreased with the value of δ. Using relative improvements allowed erasing the influence of the population mortality. For given effect sizes of X and Y, the difference in all measures of reclassification decreased when a correlation between X and Y was introduced.

Reclassification methods, particularly the minimal net reclassification improvement, seem to be clinically relevant when used with continuous clinical data with no known threshold.