A Novel Attention-based Network for Geometry Reconstruction with Error Estimation from Medical Images

Instance segmentation of anatomical structures from medical images can help enhance clinical outcomes such as disease diagnosis, surgical planning accuracy, and treatment efficacy. However, since segmentation masks often lack point-to-point correspondence between patients, instance segmentation masks may not be directly applicable for clinical studies that require measurement of medical parameters defined on an anatomical shape atlas. For such applications, meshes with correspondence between patients are preferred representations of object geometries. The conversion from segmentation masks to meshes can be error-prone due to segmentation artifacts, and therefore, it is desirable to directly obtain mesh representations of object geometries from medical image data, bypassing segmentation masks. In this work, we propose novel attention-based neural networks for geometry reconstruction and error estimation, which offers a direct pathway from medical images to high-quality mesh representations. We introduce an innovative attention-based feature extraction network and incorporate image self-attention and shape self-attention with cross-attention between them to capture consistent features. Based on the extracted features, we develop a geometry reconstruction network that deforms a mesh template to reconstruct the geometry of the object, which automatically ensures mesh correspondence. In addition, we design a shape error estimation network to evaluate the reliability of the output from geometry reconstruction, that is, to estimate the point-to-point error of a reconstructed geometry. We demonstrate our approach for the application of lumbar spine geometry reconstruction and compare our geometry reconstruction network with UNet++, UTNet, Swin UnetTR, SLT-Net, and nnUNet, using the Dice metric. In this application, our geometry reconstruction network has much higher accuracy and artifact-free segmentation results, and our shape error estimation network facilitates quality control for clinical use. The source code is available at https://github.com/linchenq/SPIE2025-GoemReconstruction-with-ShapeErrorNet.