A New Streaming K-Nearest Neighbor Algorithm for Status Prediction in Block-Sparse, Autocorrelated, Irregular...

Area of Science:

• Data Science
• Machine Learning
• Biostatistics

Background:

• Status prediction in streaming longitudinal data is difficult due to block-sparse, autocorrelated, and irregular variables.
• Existing methods struggle with such data, particularly when classes are imbalanced.

Purpose of the Study:

• To develop a robust K-Nearest Neighbor (KNN) algorithm for status prediction in streaming longitudinal data.
• To address challenges posed by data irregularity, sparsity, autocorrelation, and class imbalance.

Main Methods:

• Proposed a K-Nearest Neighbor (KNN) algorithm utilizing Kullback-Leibler (KL) divergence for distance measurement.
• Employed features from metric conditional density, with and without first-order lag.
• Developed a numerical method for distributions lacking analytical expressions, applied to real-world data.

Main Results:

• The streaming KNN algorithm achieved an Area Under the Curve (AUC) close to 1 for simulated Gaussian and inverse gamma distributions.
• The numerical method applied to a large medical streaming dataset yielded an AUC of 0.913, sensitivity of 0.851, and specificity of 0.816.

Conclusions:

• The proposed streaming KNN algorithm demonstrates high performance in status prediction for complex longitudinal data.
• The method is effective even with highly imbalanced classes and irregular data characteristics.
• The approach shows significant promise for applications in big data medical analytics.