Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration.

Area of Science:

• Computer Vision
• Medical Imaging
• 3D Reconstruction

Background:

• Differentiable rendering connects 3D scenes to 2D images, enabling learned image formation processes.
• Existing methods primarily use mesh-based 3D representations, unsuitable for medical volumetric data.
• Medical applications often utilize voxelized models for anatomical representation.

Purpose of the Study:

• To develop a novel differentiable volume rendering technique for medical applications.
• To enable end-to-end learning for 2D/3D registration using volumetric data.
• To generalize spatial transformers to projective geometry for improved 3D scene analysis.

Main Methods:

• Proposed a Projective Spatial Transformer module generalizing spatial transformers to projective geometry.
• Enabled differentiable volume rendering for voxelized 3D models.
• Applied the architecture to 2D/3D registration between radiographs and CT scans.

Main Results:

• Demonstrated end-to-end learning of an image processing and projection model.
• Achieved an image similarity function convex with respect to pose parameters for effective optimization.
• Showcased the utility of the transformer for projective transmission imaging tasks.

Conclusions:

• The Projective Spatial Transformer enables differentiable volume rendering for medical imaging.
• The method facilitates effective pose estimation and 2D/3D registration using gradient descent.
• This work advances 3D research applications in medical image analysis.