HieFormer: Leveraging hyperbolic geometry to overcome the hierarchical expressiveness limitation of transformers

3D instance segmentation requires fine-grained geometric discrimination together with an effective representation of the hierarchical organization of real-world objects and scenes. Conventional transformer architectures, however, operate predominantly in Euclidean space, whose polynomial volume growth can be poorly matched to strongly tree-like structures and can contribute to feature crowding in cluttered scenes. We propose HieFormer, whose central contribution is a unified dual-geometry design for 3D instance segmentation: hyperbolic geometry provides a hierarchy-aware space for global structural reasoning, while Euclidean geometry is retained for fine-grained local modeling, and the two are coupled within the same Transformer decoder. This design is instantiated by a learnable-curvature projection that adjusts the hyperbolic geometry during training, a hyperbolic relative position encoding (HRPE) that injects geodesic relational priors, and a Hybrid Hyperbolic and Euclidean Attention (HHEA) module that integrates hierarchical positional reasoning with discriminative Euclidean feature interactions. Extensive experiments on ScanNetv2, ScanNet200, and S3DIS show state-of-the-art or highly competitive performance. We further examine final decoder query representations under controlled structural incompleteness. When instance points are progressively removed, HieFormer exhibits a substantially more consistent ordered representation response than the Euclidean baseline, while UMAP visualization shows clearer organization across retention conditions. These findings provide representation-level evidence consistent with the intended hierarchy-aware structural modeling of HieFormer.