A New Streaming K-Nearest Neighbor Algorithm for Status Prediction in Block-Sparse, Autocorrelated, Irregular...

In streaming longitudinal data, status prediction becomes challenging when input variables are block-sparse, autocorrelated, and irregular in both dimension and distribution. General methods cannot model such data directly, especially when the classes are extremely imbalanced. This research proposes a K-Nearest Neighbor (KNN) algorithm where distance is measured by Kullback-Leibler (KL) divergence. The algorithm uses features extracted from metric conditional density, both with and without first-order lag. The developed streaming KNN algorithm is further applied to simulation data. Results show that when differences originate from the location hyperparameters of the Gaussian distribution or both the shape and scale hyperparameters of the inverse gamma distribution, the method performs quite well, as expected, with an AUC close to 1. Additionally, a numerical method is proposed for general distributions that lack an analytical expression in real data. This method is applied to a big medical streaming dataset with similar properties. Results indicate that the AUC value gradually increases to 0.913, with a sensitivity of 0.851 and a specificity of 0.816.