Finding HSP neighbors via an exact, hierarchical approach

The Half Space Proximal (HSP) graph is a low out-degree monotonic graph with a wide range of applications in various domains, including combinatorial optimization in strings, enhancing $k\text{NN}$ classification, simplifying chemical networks, estimating local intrinsic dimensionality, and generating uniform samples from skewed distributions, among others. However, the linear complexity of finding HSP neighbors of a query limits its scalability, thus motivating approximate indexing which sacrifices accuracy in favor of restricting the test to a small local neighborhood. This compromise leads to the loss of crucial long-range connections which as a result introduce false positives and exclude false negatives, and compromising some of the essential properties of the HSP. To overcome these limitations, this paper proposes a fast and exact algorithm for computing the HSP which enjoys sublinear complexity as demonstrated by extensive experimentation. Our hierarchical approach leverages the triangle inequality applied to pivots to enable efficient HSP search in metric spaces with the Hilbert Exclusion property. A key component of our approach is the concept of the shifted generalized hyperplane between two points, which allows for the invalidation of entire groups of points. Our approach ensures the computation of the exact HSP with efficiency, even for datasets containing hundreds of millions of points.