HyperNATE: Scaling tensor-based hypergraph neural networks through attention

Hypergraphs are essential for modeling higher-order interactions prevalent in complex systems, demanding specialized neural network architectures. Hypergraph Neural Networks (HGNNs) have gained popularity, yet face critical limitations. Tensor-based HGNNs (t-HGNNs), while capturing rich multi-way dependencies via tensor representations, suffer from prohibitive computational costs associated with their message passing scheme, hindering scalability. Furthermore, many HGNNs implicitly rely on homophily assumptions, limiting their effectiveness on datasets exhibiting heterophily, where connected nodes possess dissimilar features. To overcome these challenges, we introduce the Hypergraph Neighborhood Aggregation Transformer Encoder (HyperNATE), a novel neural architecture. HyperNATE decouples the computationally intensive tensor-based message aggregation from the main training loop via pre-computation. It employs a transformer encoder architecture, leveraging self-attention for parallelized multi-hop neighborhood aggregation. Crucially, HyperNATE incorporates a high-pass filter during neighborhood aggregation, enhancing its ability to capture discriminative features in heterophilic settings and mitigate oversmoothing. Empirical evaluations on node classification benchmarks demonstrate that HyperNATE achieves training speeds 10-100x faster than t-HGNNs while delivering state-of-the-art performance. Notably, it shows strong capabilities on large hypergraphs and datasets characterized by heterophily.