Real-time deformable structure tracking in 3D ultrasound sequences using deformable convolutional layers

Ultrasound imaging can provide 3D images of soft tissue structures in real-time without harmful radiation. Due to its high level of availability and low-cost characteristics, it is becoming more and more interesting for therapy guidance purposes like in radiotherapy. However, for usage in radiotherapy a robust and real-time image analysis method is required to be able to track the target during the treatment session. Soft tissue structures move in high dimensional motion patterns, including deformation especially due to breathing which makes the tracking task challenging. To overcome the deformation complexity, a novel real-time capable tracking approach in 3D ultrasound is proposed in this study. Deformable convolution layers are included in a 2D convolutional autoencoder architecture to learn deformation-invariant representations from ultrasound patches. For this, a novel 3D to 2D ultrasound patch reduction strategy is proposed. The tracking procedure is performed in the representation space of the autoencoder. We therefore implemented a greedy local search tracking algorithm and evaluated it in a preliminary study on the basis of nine expert-labeled landmarks in 18 in-vivo 3D ultrasound liver sequences. Four different autoencoder architectures with different deformable convolution arrangements are compared in the evaluation. The results show that using deformable convolution layers is beneficial compared to conventional convolution layers. A mean tracking error of 1.58 ± 0.87 mm was measured using deformable convolution, which is an improvement by 10.7% compared to conventional convolution. Additionally, evaluating the runtime shows that the tracking algorithm is real-time capable as the mean runtime per 3D ultrasound frame was around 4 ms. It could be shown that using deformable convolution layers is beneficial for learning meaningful representations of deformable structures from ultrasound patches. A tracking error similar to state-of-the-art methods could be achieved but in a runtime up to 100 times faster.