TomoGraphView: 3D medical image classification with omnidirectional slice representations and graph neural networks

Area of Science:

• Medical Imaging Analysis
• Computer Vision
• Machine Learning

Background:

• Medical tomography examinations are increasing, driving demand for automated feature extraction in 3D volumes for tasks like tumor characterization.
• Existing 3D classification methods struggle with complex spatial dependencies and limited large-scale 3D datasets, hindering progress in 3D foundation models.
• Current approaches often repurpose 2D foundation models trained on natural images, but slice-based decomposition limits their effectiveness in capturing true 3D spatial extent and coherence.

Purpose of the Study:

• To introduce TomoGraphView, a novel framework designed to overcome the limitations of traditional slice-based decomposition for 3D medical image analysis.
• To improve the accuracy and spatial coherence of feature extraction from 3D volumetric data, particularly for oncology applications.
• To provide a more effective alternative to existing methods that rely on 2D foundation models for 3D medical image classification.

Main Methods:

• Developed TomoGraphView, integrating omnidirectional volume slicing with spherical graph-based feature aggregation.
• Implemented omnidirectional slicing by sampling cross-sections from uniformly distributed points on a sphere enclosing the volume, capturing both canonical and non-canonical views.
• Utilized a graph neural network to aggregate features from these diverse viewpoints, preserving spatial relationships and 3D geometry.

Main Results:

• Omnidirectional volume slicing improved average Area Under the Receiver Operating Characteristic Curve (AUROC) from 0.7701 to 0.8154 compared to traditional slicing.
• Graph neural network-based feature aggregation further enhanced AUROC performance from 0.8198 to 0.8372.
• TomoGraphView outperformed large-scale pretrained 3D medical imaging models across six oncology datasets and various classification tasks.

Conclusions:

• TomoGraphView offers a powerful framework for volumetric analysis, significantly advancing 3D medical image classification.
• The integration of omnidirectional slicing and graph-based aggregation effectively captures complex spatial dependencies in 3D medical data.
• This approach represents a key step towards developing robust 3D foundation models for medical image analysis, bridging the gap until fully native models are available.