Net reclassification index for ordinal outcome

Abstract

In clinical research, identifying new factors that improve predictions of certain disease outcomes is important. Net reclassification improvement (NRI) is a useful measure for assessing the added predictive ability of a new factor. NRI was developed to assess improvements in diagnostic accuracy for various outcomes, including binary, survival, and multiclass outcomes. Ordinal outcomes, such as diagnosis ratings and disease stages, are also important endpoints, for which NRI has not been considered.

In the present study, application of NRI for ordinal outcomes is proposed by extending NRI for binary outcomes and multiclass outcomes with weights that take into account the closeness to the true category when counting reclassification. The standard error of the proposed NRI can be estimated utilizing the variance estimation procedures of the Stuart-Maxwell test and Bhapkar’s test statistics.

A simulation study was designed to assess the performance of the proposed method and to compare it with existing methods, such as volume under the receiver operating characteristic surface for ordinal data, reclassification index and NRI for multiclass data, and NRI and the area under the receiver operating characteristic curve for binary data with arbitrary cutoff points. Among the simulation results, the proposed method demonstrated a higher coverage rate for predictive ability than the other methods, especially Delong’s method. Also, a simulation setting based on an ordinal structure was more stable than that of a multinomial structure in regards to relative risk. The proposed method was also found to be simple and exhibited a short computing time, while Nakas’s method was complex and had a long computing time.

To validate the study results, the noted methods were applied to glaucoma data and nonrelevant cerebral atherosclerosis data. For the glaucoma data, the predictive ability of glaucomatous eyes measured by ΔAUC and NRI using Pencina’s method and Delong’s method was not improved by adding new factors to existing known factors when considering binary outcomes according to arbitrary cutoff points. Meanwhile, NRI using the newly proposed method led to improved predictive ability with the addition of new factors to existing known factors. For nonrelevant cerebral atherosclerosis data, applying NRI by the newly proposed method revealed improvement in reclassification with the addition of the presence of nonrelevant cerebral atherosclerosis or burden of nonrelevant cerebral atherosclerosis when the original categories of seven was kept; however, the significance of NRI was not shown in other categories divided arbitrarily. In other words, the predictive ability increased when keeping the original categories of the ordinal outcomes.

The newly proposed method for ordinal outcomes described in the present study is a useful discriminant measure with a short computing time and it is simple to interpret. Therefore, the method is more readily applicable to real data than other existing methods.