An adaptive and fully automatic method for estimating the 3D position of bendable instruments using endoscopic images

Flexible bendable instruments are key tools for performing surgical endoscopy. Being able to measure the 3D position of such instruments can be useful for various tasks, such as controlling automatically robotized instruments and analyzing motions.

An automatic method is proposed to infer the 3D pose of a single bending section instrument, using only the images provided by a monocular camera embedded at the tip of the endoscope. The proposed method relies on colored markers attached onto the bending section. The image of the instrument is segmented using a graph-based method and the corners of the markers are extracted by detecting the color transitions along Bézier curves fitted on edge points. These features are accurately located and then used to estimate the 3D pose of the instrument using an adaptive model that takes into account the mechanical play between the instrument and its housing channel.

The feature extraction method provides good localization of marker corners with images of the in vivo environment despite sensor saturation due to strong lighting. The RMS error on estimation of the tip position of the instrument for laboratory experiments was 2.1, 1.96, and 3.18 mm in the x, y and z directions, respectively. Qualitative analysis in the case of in vivo images shows the ability to correctly estimate the 3D position of the instrument tip during real motions.

The proposed method provides an automatic and accurate estimation of the 3D position of the tip of a bendable instrument in realistic conditions, where standard approaches fail.