Shape estimation for image-guided surgery with a highly articulated snake robot

In this paper, we present a filtering method for estimating the shape and end effector pose of a highly articulated surgical snake robot. Our algorithm introduces new kinematic models that are used in the prediction step of an extended Kalman filter whose update step incorporates measurements from a 5-DOF electromagnetic tracking sensor situated at the distal end of the robot. A single tracking sensor is sufficient for estimating the shape of the system because the robot is inherently a follow-the-leader mechanism with well defined motion characteristics. We therefore show that, with appropriate steering motion, the state of the filter is fully observable. The goal of our shape estimation algorithm is to create a more accurate and representative 3D rendered visualization for image-guided surgery. We demonstrate the feasibility of our method with results from an animal experiment in which our shape and pose estimate was used as feedback in a control scheme that semi-autonomously drove the robot along the epicardial surface of a porcine heart.


I. INTRODUCTION
With minimally invasive surgery (MIS), a physician typically performs diagnostic or interventional procedures with a surgical tool or robot through small port-like incisions in order to reduce patient trauma. Unfortunately, with MIS, surgeons cannot view an operation with direct vision and instead must rely on indirect imaging for surgical guidance. It is common for a surgeon to use fluoroscopy [1], ultrasound [2], MRI [3], or endoscopy [4] for this purpose. Unfortunately, all of these modalities have limitations. Another option is imageguided surgery, which seeks to display a virtualized rendered view of an operation for guidance by fusing information from a tracking device with preoperative surface models [5].
In this paper, we present a nonlinear stochastic filtering method that estimates, with measurements from a magnetic tracking sensor, the shape and configuration of a high degree of freedom surgical snake robot, see Fig. 1. The goal of this work is to display an accurate rendering of the snake robot alongside preoperative surface models for image-guidance. While it is possible to simply track the position of the distal tip of the robot, we instead believe that estimating the full shape and configuration of the robot would provide more informative feedback to the surgeon: e.g., if a trajectory to an anatomical target fails due to an anatomical obstruction, viewing the full shape of the robot in relation to the anatomy would tell the surgeon how and why the intended path failed.  Another reason for estimating the full shape is that we can infer twist in the robot's configuration, which can be useful for righting joystick inputs and rectifying video. An example of the image-guidance we achieve is shown in Fig. 2.
To perform shape estimation for a cable-driven snake robot, we use an extended Kalman filter (EKF) formulation with newly defined motion models and a forward kinematic measurement model that incorporates 5-DOF pose measurements at the distal tip of the robot from an electromagnetic tracking sensor. Full shape estimation is possible, in this context, because the robot is inherently a follow-the-leader device with explicitly definable motion models.
The contributions of the work presented in this paper are 1) the novel use of an EKF to estimate the shape of a surgical snake robot, 2) new motion models for a cable-driven surgical snake robot, 3) an analysis of the observability of shape estimation with a single 5-DOF tracking sensor at the tip of the robot, and 4) a feasibility study of our shape estimation method through the discussion of an experiment that involved navigating a robot semi-autonomously on the epicardial surface of a porcine heart.

A. Image-Guided Surgery
Image-guided surgery is a term that is often used to describe a procedure that uses patient-specific medical images as a form of visual feedback during surgery. In many cases, this equates to using preoperative CT or MRI data to reconstruct a 3D surface model of anatomical structures, as in [6]. With image-guided surgery, a tracking device is integrated with a surgical tool and registered to the preoperative images so that the position of the tool can be overlayed on the rendered models, as in Fig. 2.
An example of image-guidance is [5], in which Cleary et. al. use an electromagnetic (EM) tracker registered with preoperative images. Also, in [7], an automatic registration method is introduced to align EM tracker measurements with preoperative images using an iterative closest point (ICP) method. Commercial examples include Ensite NavX (St Jude Medical, St Paul, MN, USA) and Carto XP/CartoMerge (Bio-Sense Webster, Diamond Bar, CA, USA), which have been applied successfully to electrical mapping for cardiac ablation. The majority of existing methods track the tip of a surgical tool in real-time, but we believe it would be more informative to view the entire configuration of the tool, as in Fig. 2. Tracking the full shape is the subject of this paper.

B. Shape Estimation
The use of Fiber Bragg Grating (FBG) sensors is becoming a popular method for estimating the shape of a flexible tool. For example, in [8], the authors use an optical fiber with FBG sensing to determine in real-time the shape of a colonoscope. Likewise, in [9], a novel slim FBG wire is inserted into the biopsy channel of a colonoscope to determine shape. In [10], the authors use optical FBG strain-sensors to measure the shape of a flexible needle in the field of an MRI. While this is one of the more popular methods for computing the shape of a tool, there are several issues: the first is that the sensor is temperature dependent. The second issue is that, while the overall shape may be accurately detected, the Euclidean position at the end effector may have accumulated error. Our shape estimation algorithm presented in this paper avoids these two drawbacks.

C. HARP Surgical Robot
The robot we are using for MIS is a highly articulated robotic probe (HARP), which is a surgical snake robot presented in [11]- [13]. The advantage of the HARP is that it has the stability of a rigid device as well as the maneuverability of a flexible tool (a photograph of the robot can be seen in Fig. 1). This type of robot is unique, in that it can navigate any curve in a three-dimensional space with only six actuators. The HARP is made up of many rigid links which are actuated at the distal end by three cables. A prototype version of the HARP has been used experimentally to investigate epicardial ablation on porcine models [13].

III. SNAKE SHAPE ESTIMATION
The objective of our snake shape estimation method is to recursively compute the most likely state parameters that define the robot's shape and configuration during image-guided MIS. In this section, we will define the state vector that we are estimating as well as the motion and measurement models that we have developed for this filtering problem.

A. Kalman State Definition
The state that we are estimating in a Kalman filter framework is defined as follows, An example of overlaying a model of a surgical robot on preoperative surface models for image-guidance. This is a live snapshot from an experiment with the HARP navigating semi-autonomously on the epicardial surface of a porcine heart.
where (x 0 , y 0 , z 0 ) is defined to be the position of the most proximally located link of the robot that we are interested in tracking at time-step k. There typically will be links behind this first link that we do not care about until they advance forward, see Fig. 3. Also, (α 0 , β 0 , γ 0 ) are the yaw, pitch, and roll respectively of that first link. The terms φ i and θ i for each i are angle offsets that we will discuss shortly. To help formulate the pose of a rigid body in three dimensions, we define the following three rotation matrices, where the trigonometric notation has been simplified for convenience (i.e., s γ = sin(γ)). With these rotation matrices, we can describe the pose of the most proximally referenced link as a function of the Kalman state with a transformation matrix, The pose of more distally located links are also defined by the state vector as follows: the elements φ i and θ i in the Kalman state definition from Eq. 1 are offset angles associated with link i that define link i's orientation relative to the preceding link. A visual interpretation of φ i and θ i can be seen in Fig. 4.
To compute the transformation matrix T i (X k ) that represents the pose of link i, we define the following recursive process that is initialized with the pose of the starting link, where L is the length of a link. As seen in Fig. 3, each link i has an associated transformation matrix that can be computed from the previous transformation matrix via the offsets φ i and θ i . Thus, the state vector from Eq. 1 sufficiently defines the pose of all links and thus the shape and configuration of the robot.

B. Advancing Motion
The HARP is a multi-link robot that is, by design, a follow-the-leader device. (see [11] for the mechanism design details). The robot maintains its shape in three-dimensional space and when commanded, advances one link length at a time: each link theoretically moves into the corresponding pose of the link in front of it. In this case, a link behind the most proximally referenced link will move into its place and assume the role of the starting link of the Kalman state with transformation matrix T 0 . The way the robot advances can be seen in Fig. 5. When all of the links advance one step ahead, the state space grows by two parameters (there is effectively one extra link in the state), as seen in Fig. 5. The motion model for this advancing step can be defined with the following function,

C. Retracting Motion
Like advancing, when commanded to retract, the HARP maintains its shape in three-dimensional space. The most proximally referenced link moves backwards into a position that is not tracked by the Kalman state vector while the link one step ahead moves into its place and assumes the role of the starting link of the Kalman state with transformation matrix T 0 . The distal link also theoretically moves into the position of the link preceding it. Assuming M is the length of the state vector at time-step k, the motion model for retracting is, The length of the state is reduced by two because the number of links tracked by the Kalman state is reduced by one.

D. Steering Motion
When the HARP is in steering mode, all of the links preceding the distal link in the state space are restricted from moving because an inner mechanism is locked into assuming the current shape (see [11] for details). This means that actuating the three cables that run through the entirety of the snake will theoretically control just the orientation of the distal link. Thus, by pulling on these three cables in different amounts with the proximally mounted motors, the pose of the distal link will change.
We have formulated a steering model that determines the angle offsets φ N−1 and θ N−1 of the distal link relative to the link preceding it based on the cable lengths, where N is the number of links we are tracking in the Kalman state vector, .
For this model, C R is a radius term that depends on the separation of the cables and (c 1 , c 2 ) are the measured differential lengths of each of two cables running down the robot, relative to the positions that the cables were in after advancing. The value c 3 associated with the third steering cable in the robot does not appear in this model because it is geometrically a function of c 1 and c 2 and is therefore redundant information.
The derivation of this model is omitted for brevity but we note that it is based completely on the geometry of the distal link of the robot. An interpretation of this steering model is as follows: 1) the angle at which the link will be oriented depends on which cables you pull tighter and 2) the extent that the link will be angled depends on the amount we pull on the cables. We again refer to Fig. 4 for a depiction of the angles that are affected by the actuation of the cables. From the measured cable lengths, which are obtained from encoders on the actuated motors, we can obtain the new angle offsets φ N−1 and θ N−1 of the distal link of the robot using Eqs. 4 and 5. We use these updated values to compute the change in angles from the previous time step, stored as ∆φ and ∆θ, and then formulate the following motion model for steering the HARP,

E. Measurement Model
The sensor we are using for image-guidance is a magnetic tracking sensor situated at the distal end of the snake robot. The device we are using is the trakSTAR TM (Ascension Technologies, Burlington, VT, USA), which can measure the 6-DOF pose of a sensor in three-dimensional space. We insert the tracker into one of the tool channels of the HARP.
While the tracker is designed for 6-DOF pose estimation, only 5 degrees of freedom are useful in our application. This is because the tracker must be inserted into the HARP such that it can be removed easily for exchanging tools, and thus the roll parameter of the tracker is free to rotate within the working channel. The measurement therefore directly observes five elements of the pose of the distal link of the robot, and we can formulate the measurement model as, where p N−1 is the position of the distal link, as in Eq. 2, and (α N−1 , β N−1 ) are the yaw and pitch of the distal link. All of these parameters can be extracted from T N−1 (X k ), which is computed from the state X k .

F. EKF Formulation
In this paper we are introducing a method to estimate the state of the HARP given the measurements obtained at the distal tip by a magnetic tracker. Because we have well defined motion models and a forward kinematic measurement model, it is reasonable to formulate this filtering task in the framework of a Kalman filter (specifically an extended Kalman filter because of nonlinear models). The purpose of using a filter to estimate the state is that the motion and measurements are subject to noise and disturbances.
The first step of our EKF formulation is to initialize the estimate of the state. To do this, we begin an experiment with the snake robot completely retracted with the magnetic tracker in the distal link, which also happens to be the only link in the Kalman state. A depiction of the state of the system is shown in Fig. 6-(a). In this situation, a single magnetic tracker measurement directly measures the 5-DOF pose of the first link in the Kalman state. We can therefore initialize the mean and covariance matrix of our EKF implementation as follows, where z 0 is the initial sensor measurement which is modeled according to Eq. 6. The roll parameter in the initialized mean is set to zero because we do not yet have enough information to set a value for this element and thus we must initialize 6. This shows the first two steps of our initialization process for the Kalman filter.
the roll arbitrarily. For the covariance initialization, R is the uncertainty in the sensor noise and σ 2 γ is a variance value chosen by the user that models the large initial uncertainty in the roll parameter of the state.
After this first measurement, we advance the robot one step and evolve the mean of the filter based on the motion model in Eq. 3. As for the covariance, we add a small amount of noise to represent the fact that parameters may be disturbed through the actuation of the cables. The state of the robot after advancing is depicted in Fig. 6-(b). The new estimate becomes (X 1|0 ,P 1|0 ). The reason for advancing the robot an extra step before any steering takes place is that it simplifies the formulation of our filtering method because our steering model from Sec. III-D is defined for at least two links. After the robot advances, another measurement is acquired from the magnetic tracking sensor and the standard Kalman measurement update is applied using the measurement model in Eq. 6. The new estimate then becomes (X 1|1 ,P 1|1 ).
After this initialization procedure, we can subsequently rely on the motion and measurement models defined in this section to predict and update the EKF in real-time using the well known Kalman prediction and update equations. We note that we add prediction noise (to the variances of the link angles) after each steering command. One difference between our filtering scheme and a conventional EKF implementation is that the prediction step that we perform at any given timestep will depend on the mode that the robot is in (steering, advancing, or retracting). The overall algorithm for our EKF implementation is described in Alg. 1. 3: if mode = steer then 4:

IV. OBSERVABILITY OF SHAPE ESTIMATION
To achieve shape estimation, we are estimating the joint angles of a high degree of freedom snake robot with only a magnetic tracker that measures the pose at the distal tip of the robot. To support our claim that this methodology is sufficient for shape estimation, we will introduce here an analysis of the observability of the filtering problem defined in Sec. III. Observability is a measure of whether the state of a system can be obtained from the system outputs (measurements) [14]. For this analysis, we are using linearized models to construct the observability matrix because it is sufficient for revealing the conditions under which successful estimation of shape for the HARP is possible.
As discussed in Sec. III-F, we initialize the Kalman estimate when the robot is completely retracted. At this point in time, we receive an initial sensor measurement, z 0 . Based on the measurement model in Eq. 6, we can define the initial observability matrix O 0 as follows, which has a rank of only 5 when the length of the state is 6.
As expected, the roll parameter is not observable with this initial measurement. According to our initialization procedure defined in Sec. III-F, we then advance the snake one link forward. Due to the inherent design of the snake robot, we know that the values θ 1 and φ 1 will be equal to zero when there is yet to be actuation from steering. Thus, we can treat our knowledge of these two values as a hypothetical measurement with the model h adv (X k ) = [θ N−1 , φ N−1 ] T . A new observability matrix can be written as follows, At this point, the Kalman filter has 8 parameters in the state vector, but the rank of the observability matrix is only 7.
The roll of the initial link is still not observable given the measurements we have obtained. Finally, after the robot has been initialized, the second link can be steered by actuating the cables of the robot. The motion model in Sec. III-D defines the orientation change of the distal link when a steering command [∆φ, ∆θ] is applied to the system. After steering, a third component can be added to the observability matrix, where H 2 and F 2 are linearized Jacobians, The rank of O 2 is equal to 8 as long as ∆φ is unequal to zero. The significance of this analysis is that as long as we steer the robot after advancing the link, we can observe the roll of the system. This is expected because by moving the distal link so that it is off axis, we can observe the direction Ground Truth Estimate Fig. 7. Data from an experiment we performed in the lab, for which we recorded ground truth shape data (solid line) to compare with our shape estimation algorithm. The average error for a link location was 2.98mm.
in which the robot steers and thus we can determine the orientation of the coordinate frame of the robot. Lastly, once the 8 parameters that define the state of the first two links are observed, the motion models will precisely define the evolution of the state based on motion inputs. Thus, according to the models we present in this paper, the entire shape of the robot is fully observable.
While this is a significant result for supporting the use of this filtering method, the observability analysis we present here assumes perfect models in which the robot is truly a follow-the-leader device that maintains its shape. In real life, though, the shape estimate may be affected by noise and external forces. Thus, to evaluate the realistic performance of our filtering scheme, we will discuss two experiments in the next section.

A. Experiment I
The first experiment that we will discuss is a benchtop test in which we drove the snake robot in an S-curve configuration, turning maximally to the right and then maximally to the left. The shape of the curve can be seen in Fig. 7. The magnetic tracker (trakSTAR TM from Ascension Technologies, Burlington, VT, USA) remained at the tip of the robot throughout the experiment and was used to update the shape of the robot with the filtering algorithm in Sec. III.
At the end of the experiment, we fixed the shape of the snake and subsequently pulled the magnetic tracker through the working channel to record a trail of data points that we could post-process as a ground truth path. We compared our shape estimate to the ground truth with an average error of 2.98mm and a maximum error of 6.74mm between the robot links and the corresponding nearest point on the ground truth path. The result is shown in Fig. 7. We attribute the accuracy of this result to our estimation algorithm as well as the robot's inherent ability to preserve its shape as a followthe-leader device. We note that, while this is a promising outcome, we believe the algorithm has the potential for more accurate estimation. We argue that additional tuning of noise parameters could improve performance.

B. Experiment II
The second experiment we performed (Experiment II) involves a live animal experiment in which we tested our own image-guidance software while navigating the HARP robot along the epicardial surface of a porcine subject. One of the goals of this experiment was to investigate the feasibility of performing epicardial ablation with the HARP robot with a subxiphoid approach. In Fig. 8, we show the HARP next to the single-port incision. We note that the photo in Fig. 8 was taken during a previous experiment.
During Experiment II, we operated the robot semiautonomously, which meant that the robot steered itself along a prescribed path to a target location. For this experiment, we tested our shape estimation algorithm and show the qualitative result in Fig. 9. Although we unfortunately do not have ground truth data for this experiment, we have reason to believe that the filtering algorithm we are presenting in this paper performed well in estimating the shape during this animal trial because the resulting images show the robot correctly configured in just the right shape so as to lie between the surface of the heart while also lying beneath the rib cage of the porcine subject.

VI. CONCLUSION
The contribution of the work presented here is a novel filtering method for estimating in real-time the shape and pose of a highly articulated surgical snake robot. Our algorithm combines new kinematic models of the motion of the robot with measurements from a magnetic tracking sensor in a custom EKF framework. We have shown that the system is fully observable under appropriate motion, which supports the use of this algorithm in experiments.
The advantage of the proposed approach is that we can leverage existing magnetic tracking technology that would already be used for estimating the pose at the tip of the surgical robot to also determine the full configuration of the snake. In this case, the determination of shape is important for implementing a fully representative 3D visualization.
We have shown promising results, both with bench-top testing and animal experiments. In one experiment, the HARP was driven semi-autonomously along a predefined path using feedback from our EKF implementation. This is an exciting result, for it demonstrates both the capabilities of the robot as well as the ability of our algorithm to accurately filter the configuration of the robot in real-time.
An additional benefit of our shape estimation algorithm is that it can be used to determine if there is any twist along the length of the HARP. Extracting this information in realtime during an operation can help rectify video and right the joystick inputs for more intuitive control of the robot.
Unfortunately, it is worth noting that if the robot is interacting with deformable tissue and/or moving tissue (in the case of a cardiac experiment), the estimation of shape may be adversely affected. For this problem, more advanced models are required, which is the subject of future work. Fig. 9. This is a result from an experiment in which we semi-autonomously navigated the HARP along the epicardial surface of a porcine heart and used our shape estimation algorithm for visual feedback.