Failure Recovery in Robot–Human Object Handover

Object handover is a common physical interaction between humans. It is thus also of significant interest for human–robot interaction. In this paper, we are focused on robot-to-human object handover. The main challenge in this case is how to reduce the failure rate, i.e., to ensure that the object does not fall (object safety), while at the same time allowing the human to easily acquire the object (smoothness). To endow the robot with a failure recovery mechanism, we investigated how humans detect failure during the transfer phase of the handover. We conducted a human study that showed that a human giver primarily relies on vision rather than haptic sensing to detect the fall of the object. Motivated by this study, a robotic handover system is proposed that consists of a motion sensor attached to the robot's gripper, a force sensor at the base of the gripper, and a controller that is capable of regrasping the object if it starts falling. The proposed system is implemented on a Baxter robot and is shown to achieve a smooth and safe handover.

Failure Recovery in Robot-Human Object Handover Sina Parastegari , Ehsan Noohi , Bahareh Abbasi, and Miloš Žefran Abstract-Object handover is a common physical interaction between humans.It is thus also of significant interest for humanrobot interaction.In this paper, we are focused on robot-to-human object handover.The main challenge in this case is how to reduce the failure rate, i.e., to ensure that the object does not fall (object safety), while at the same time allowing the human to easily acquire the object (smoothness).To endow the robot with a failure recovery mechanism, we investigated how humans detect failure during the transfer phase of the handover.We conducted a human study that showed that a human giver primarily relies on vision rather than haptic sensing to detect the fall of the object.Motivated by this study, a robotic handover system is proposed that consists of a motion sensor attached to the robot's gripper, a force sensor at the base of the gripper, and a controller that is capable of regrasping the object if it starts falling.The proposed system is implemented on a Baxter robot and is shown to achieve a smooth and safe handover.

I. INTRODUCTION
O BJECT handover is a common type of interaction between humans.In a handover, a giver hands the object off to a receiver.While they both participate in the exchange, they each have a different goal: the giver wants to safely release the object, whereas the receiver wants to readily acquire the control of the object and establish a stable grasp [1].
In a handover task, the most critical phase is the object transfer phase, in which the object load is gradually transferred from the giver to the receiver [2].The transition starts from the time that the giver makes the decision to open her hand and release the object, and it ends when the object is fully released and the giver is no more in contact with the object.The timing needs to be precisely coordinated between the giver and the receiver.On the giver's side, releasing the object too early may result in the object falling, while releasing it too late results in high interaction forces [2].
Humans highly benefit from different mechanisms to prevent failure during a handover.Since the focus of this paper is robot-to-human handover, we are primarily interested in what the giver can do.A human giver may attempt to catch the object before it hits the ground.In addition, using social-cognitive reasoning based on haptic information, gaze, the pose of the receiver's body, and the configuration of her hand, humans have a remarkable ability to judge whether the receiver is ready to grasp the object during the handover.In this regard, it is worthwhile noting that haptic interaction during a handover has been shown to be substantial [2].In particular, after the receiver has reached the object and before the object transfer phase is initiated, if the haptic sensing suggests that there is a problem, the giver will never release the object and a potential failure will be avoided.
Object handover has been widely studied in the human-robot interaction literature as a typical collaboration between humans and robots.Researchers have studied how to perform humanlike handover [3]- [6] and how to implement smooth and safe robot-human handover controllers [7]- [10].While the performance of the robot-human handover controllers has steadily improved, the failure rate (the object is dropped) of the proposed controllers is still high compared to human-human handovers [2].Failure detection mechanisms that humans possess are a challenge for robots: their bulk prevents them from performing quick movements needed to catch the object, and their ability to interpret social-cognitive cues from different sensors is still limited.But compared to humans, they can process sensory data and react to it much faster.Our goal in this paper is thus to investigate how robots can take advantage of their processing speed to compensate for what they lack in agility and cognition.In particular, we will show that when the robot is the giver, by judiciously choosing the sensors used for detecting an impending fall of the object, it can prevent failure by closing its gripper before the object falls out of reach.This mechanism also allows the robot to handle collisions during the handover, which are another significant source of failure for existing handover controllers.
In the first part of this paper, we investigate how human givers detect a failure during the transfer phase of the handover.The main question that we want to answer is which sensory modalities are used by human givers to detect a failure?We describe a human study designed to investigate this question.The human study shows that humans primarily rely on vision; that is they detect the impending fall of the object by observing its motion.This finding is then used in the later part of this paper to design a robot handover system that consists of a motion sensor attached to the robot's gripper, a force sensor at the base of the gripper, and a controller that is capable of regrasping the object if it starts falling.We show that the proposed system achieves a safe and smooth handover.
The idea of using object's motion information to achieve a safer handover has been explored in the past.In [9], the object acceleration is continuously measured and the controller ensures that the acceleration is smaller than a threshold before releasing the object.While this method improves the object safety (prevents drops), it cannot recover from a failure once the object has been released, e.g., if the object slips from the receiver's hand.Also, it cannot handle moderate collisions between the receiver's hand and the object.In this paper, we propose a handover scheme for a two-finger robotic gripper that is able to detect an impending failure based on the object acceleration and recover from it by regrasping the object.The system was introduced in our preliminary work [11], but no analysis was provided regarding its performance under various task conditions.Our contribution here is threefold: first, we present a human study that shows that human givers mainly rely on the object's motion to detect a failure during the handover transfer phase and that informs the choice of sensors used in our implementation.Second, we describe a framework for a fail-safe handover controller that can detect and prevent a failure.Finally, a simulation is presented that shows that the system is robust with respect to the sensor noise, actuator disturbances, and model uncertainties.The proposed handover system is implemented on a Baxter robot and it is shown that it can effectively prevent a failure, it allows the receiver to take the object at a wide range of angles, and it requires a smaller pulling force compared to the existing handover controllers.

II. RELATED WORK
Different characteristics of human-human handovers have been studied in the literature.Shibata et al. [12] analyzed the human hand trajectory during the task.Basili et al. [13] studied reaching motion of humans and Mason et al. [1] investigated grip forces applied to the object by the giver and the receiver.
Many of the proposed handover controllers for robots are inspired by human-human handovers.Kajikawa et al. [7] presented a method to generate humanlike motion for the robot to perform the handover.Hendrich et al. [8] presented a multimodal robot-human handover controller which tracks the human motion and detects the beginning of the handover by combining clues such as visual signals, sounds, etc.Other works include investigating how to approach a person for a handover [3], how to choose the location of the handover [4], and how to present the object to the receiver [6].In [14], we proposed a model to select the handover configuration and the reaching motion trajectory based on what humans do in humanhuman handovers.
Strabala et al. [15] proposed a coordination architecture for the robot-human handover in which three main questions are addressed: what, when, and where.Answering these questions results in initializing the handover task, establishes the timing of the handover, and specifies the configuration in which the handover will occur.
Answering the when question is one of the most important issues in robot-human handovers [15].In case that the robot is the giver, there should be an accurate plan when to approach the receiver.The robot also has to open its hand and release the object at the right time.Several methods have been proposed regarding when to release the object.A simple algorithm is to release the object after a predefined period of time [5], [16].This algorithm frequently fails by dropping the object since there is no coordination between the robot and the receiver.Deyle et al. [17] proposed an algorithm in which the robot releases the object once the applied force, measured at the wrist, exceeds a predefined threshold.Bohren et al. [18] implemented an impedance controller for the robot, so the object is released whenever the displacement of the robot's hand due to the exerted force by the receiver is more than a threshold.Another approach proposed by Nagata et al. [10] is to continuously check the stable grasp condition.The object is released once the grasp becomes unstable.This approach requires force/torque sensors at fingertips.
While the aforementioned approaches focus on a smooth object transfer, they suffer from a high failure rate.A failure happens when the object is dropped during the handover, often because of a collision with the receiver's hand or an obstacle.Due to such a collision, a similar force or displacement is measured in the robot's hand as if the receiver was pulling the object.So the robot mistakenly releases the object.To solve this issue, the controller's robustness to disturbances can be increased.But as a result, a large force must be applied by the receiver to take the object.In other words, there is a tradeoff between the handover smoothness and the object safety [9].
In another study, Chan et al. [2] investigated the relation between the load forces versus the grip forces during humanhuman handovers and showed that the grip force exerted by human givers has a linear relationship with the vertical load force.Based on this observation, a human-inspired (HI) handover controller was proposed and implemented for PR2 robot which shows smooth performance compared to other approaches [9].While the proposed algorithm has been shown to be robust against collisions with the robot's arm, it only works when the object is transferred in a vertical direction and the handover is quasi-static.
In an effort to prevent falling of the object when the object is grasped by a robotic hand, different slip detection and recovery methods have been proposed.Vibration sensing [19], optical tracking of the object [20], [21], and normal and shear stress sensing [22] are among the proposed slip detection methods.In [23], tactile and force sensors are used to detect a slip.In a study in neurophysiology [24], it is shown that humans detect slippage of objects based on the firing activity sensed by high frequency sensors in the fingertips.Once the slip is detected, the grasp force should be regulated to restrain the object.Cipriani et al. [25] proposed different hierarchical control strategies to regulate the grasp force and Yussof et al. [26] analyzed performance of a tactile-based slippage control algorithm for a robotic hand that performs grasp-move-twist motion.The aforementioned methods effectively improve the safety of the object in a grasp.However, they cannot be employed by a giver in a handover because in the object transfer phase, there might be slippage when the object is being released by the giver; this is exactly the opposite from grasping.To prevent handover failure, the controller should thus distinguish between undesired slippage caused for example by a collision versus desired slippage caused by the receiver pulling the object.

III. HANDOVER FAILURE DETECTION
In a handover, the decision to release the object is made by the giver once she is confident that the receiver has grasped the object or is capable of doing so.The decision is based on the information provided by the sensors, such as vision and haptic sensing, but it is also affected by social-cognitive processes.In [2], it is shown that during the object transfer phase, the object load is gradually transferred from the giver to the receiver.The total duration of the object transfer phase (t transfer ) is also measured and reported to be between 300 to 700 ms, in an experiment where a baton shaped object with variable weight of 480 to 680 g was handed over between the participants.Fig. 1 shows a simulation of the forces during the transfer phase of the handover that mimic those presented in [2].A simple open-loop controller was used to generate the simulation.
There are different ways a handover can fail.For example, if the giver decides to abort the handover because she determines that the receiver is not ready, this could be considered a failure.In this paper, by a failure we mean a very specific situation when the giver has made the decision to open her hand and let the object go, but for some reason, the receiver fails to grasp the object so the object starts to fall.In particular, we are interested in how the giver can detect a failure and react to it by regrasping the object.Our motivation for studying how humans detect failure during the handover is to use what we learn to design a system that allows a robot giver to recover from failures.In particular, we need to determine which sensors should be used by the robot.
We should stress that the focus of this paper is on handovers where the forces during the transfer phase follow Fig. 1.This happens for instance when the giver holds the object by opposing fingers pressing against vertical faces of the object (see Fig. 4).In this case, a release of the object immediately results in a fall.In other handover configurations, such as when passing a plate to another person, after the giver releases the object, it is still partially supported by her hand so the fall does not happen immediately and there is more time to react to a failure.
The only two senses that might help the giver to detect a failure are vision and haptic sensing (after the fact, a failure can be detected by hearing the object hit the ground; but at that point it is too late to react).Humans can always detect a failure through vision: as they follow the trajectory of the object, if the object is not in the receiver's hand, which means a failure has occurred.However, the contribution of haptic sensing is unclear.Haptic sensing is shown to play an important role in object grasping [27] and in particular in slip detection when an object is fully grasped [24].Slip detection can thus be used to control the grasp [20], [23], [26], [28].But in a handover, in contrast to grasping, it is expected that the object will slip from the giver's hand as it is transferred from the giver to the receiver so the slip does not indicate a failure.The next section describes a human study that was performed to determine whether humans in the role of a giver use haptic sensing to detect a failure during a handover. 1

IV. HUMAN STUDY
The human study is conducted with 10 participants (5 men and 5 women) between ages 21 to 37. The participants are recruited from students and staff at the University of Illinois at Chicago (UIC) by posting flyers and sending e-mails to UIC graduate students' e-mail list, according to our approved IRB protocol.The subjects play the role of the giver in the handover task by keeping an object in their hand.The experimenter plays the role of the receiver and tries to take the object from the subject.

A. Experimental Setup and Procedure
While blindfolded, each subject performs the handover task 10 times.The object to be handed over is an 8 cm × 8 cm × 9.5 cm mug (see Fig. 2).In each trial, the participant keeps the mug in her hand and the experimenter attempts to take the mug.In half of the trials (five trials), the experimenter performs the taking, i.e., the object is completely grasped using three fingers: the thumb, the index, and the middle finger.The mug is then pulled horizontally out of the participant's hand [see Fig. 3(a), the direction of pulling is shown by a blue arrow].In the other five trials, the experimenter performs the dropping, i.e., the object is intentionally dropped by putting the index and middle fingers inside the mug and pulling it horizontally [see Fig. 3(b), the direction of pulling is shown by a blue arrow].In order not to give any clue to the subject, the experimenter tries to touch and pull the mug softly so the participant cannot realize how many fingers are used.The taking and dropping actions are interleaved randomly so the participant is neither aware of the number of repetitions of each action nor the order of the actions.There is a pillow on the table so in case the object falls, it will not make a loud sound.Also, a loud music is played for the participant through headphones, so she cannot hear the sound of the object hitting the pillow.
The participant's role is to tell whether the object is dropped or taken after each trial.The expectation is that if haptic sensing contributes significantly to detecting a failure during the transfer by the giver, the participants should be able to detect the drop at a rate significantly better than chance (which would be 50%).Note that the participants are instructed not to regrasp the object if they feel it was dropped.To determine the effect of the direction of pulling, the experiment is repeated again, but with a different pulling direction.This time, the object is pulled diagonally [see Fig. 3(c) and (d), the direction of pulling is shown by a red arrow].In total, each subject thus performs 20 trials.
In order to maintain the consistency between the trials, we attached a 9-DOF Sensor-Stick SparkFun Inertial Measurement Unit (IMU) [29] to the mug to measure its acceleration (see Fig. 2).The IMU is interfaced with a computer through an Arduino MEGA development kit [30].For each trial, we make sure that the object's acceleration in the direction of pulling falls in the range of 1.5 to 4.5 m/s 2 ; otherwise the trial is disregarded.This range of accelerations was observed in a pilot study in which several subjects simply handed over the same mug to each other.The object and the IMU weigh 420± 10 g in total.
Note that during the human study, several precautions are taken to eliminate the human social-cognitive reasoning that typically takes place during the handover: the experiment is repeated several times in the same configuration and the giver cannot see the experimenter (the receiver) or her hand.

B. Results
The range of the magnitudes of the object acceleration in the direction of pulling is shown in Table I.
The performance of the participants is summarized in Table II.For example, the first row in the table indicates that when the experimenter performed the taking action and pulled the object horizontally, the success rate was 62%, i.e., in 62% of the trials, participants could correctly identify the action and express that the object was taken.The third row in the table shows that the total success rate of the participants, when the object was pulled horizontally, was 54%.In order to determine the role of haptic sensing, we used the data to test whether or not participants were able to answer correctly significantly more often than chance (50%).
Let x i , i = 1, . . ., 10, denote the success rate of each participant when the object is pulled horizontally (one sample).The average of all the samples is xi = 54% and its standard deviation is SD(x i ) = 13%.One-sample t-test indicates that the performance of the participants does not significantly differ from the 50% success rate expected by chance: t = 0.973, p − value = 0.356.To make sure that the underlying distribution assumptions of t-test do not affect the analysis, the significance level is also calculated using a nonparametric test.We use a binomial test [31] on the overall success rate of all the participants, tested against a null of 50% success rate.Using binomial test makes sense here because the action is randomly selected in each trial and hence the trials are statistically independent.This test also shows that the success rate does not significantly differ from chance: p − value = 0.481.Furthermore, the 95% confidence interval of the success rate is [0.42, 0.65] which shows that the correct and the incorrect answers are almost equally likely to be selected.
The conclusion is that in this setup, the information provided by haptic sensing is not reliable enough to detect falling of an object during the object transfer phase.In fact, the participants stated that they often could not decide with certainty whether the fall occurred and they just chose a random answer.The performance of participants is slightly improved when the object was pulled diagonally, with participants answering correctly in 67% of trials (see row 6 in Table II).We speculate that this improvement is because of the greater difference between the object's motion in taking and dropping scenarios when the object is pulled diagonally.Even though the success rate in this case is significantly higher than chance (p − value < 0.01), the failure rate of 33% is still quite high.Considering that humans can easily detect the failure with their eyes open, the data show that closing the participants' eyes has significantly affected their performance.

V. ROBOT HANDOVER SYSTEM WITH FAILURE RECOVERY
Given the results of the human study, we were motivated to choose vision instead of a haptic sensor to detect the fall of the object.In a human-human handover, vision clearly plays a significant role.The giver watches the receiver reaching for the object and perceives her readiness to take it.This information helps the giver to coordinate the time, the location, and the configuration of the handover.On the other hand, during the transfer phase, both the giver and the receiver visually monitor the object to ensure the successful completion of the task.However, visual processing is time consuming, both for humans and for computers.In order to use the motion of the object for failure detection, and to allow the robot to react to the failure, motion needs to be monitored with a sensor that has a fast response time.
Among the characteristics of the object's motion, the acceleration can be used as an indicator of an impending fall of the object and this is the modality we used in this paper.Object acceleration can be measured by installing an accelerometer on the object; however, this is clearly not a solution that is suitable for implementation on a robot.Instead, the robot should be equipped with a sensor that can measure the object's acceleration with respect to the robot's hand.We propose to attach an optical sensor similar to what is used in an optical mouse, to the finger of the robot.Optical sensors are able to measure acceleration up to 10 g and are fast [32].They often have a limited range of operation, i.e., the object should be in close proximity to the sensor (distance <4 mm).But in our application in which the object is in contact with the robot's finger, these optical sensors provide a cost-effective and practical solution.Also, like any other position-based estimate, our acceleration measurement is subject to high-frequency noise.But in Section VI, we show that our proposed controller is highly robust against acceleration measurement noise.
In the following, a model of a handover system with a twofinger robotic gripper is proposed.Subsequently, we design a handover controller that includes a regrasping mechanism.The regrasping mechanism relies on the feedback that includes the acceleration of the object measured by an optical sensor installed on the gripper, as well as forces measured at the wrist.

A. System Model
Assume a two-finger robotic gripper, grasping an object.The robot's wrist is equipped with a force sensor that measures the forces applied to the object and there is an optical sensor attached to the gripper that measures the object acceleration relative to the gripper.A human subject tries to take the object from the gripper by pulling on it.In Fig. 4, the gripper, the object, and forces applied to the object are shown.
F p is the pulling force applied by the human, ϕ is the angle of pulling, W is the object weight, F G is the controlled grip force (each finger of the gripper applies F G to the object), and F f is  the friction force between the gripper and the object.For the sake of simplicity, it is assumed that F p lies in the plane that is perpendicular to the grip force F G .Based on the configuration of the object and the gripper, the system can be in one of the three modes: the grasp mode, the slipping mode, and the release mode.Different system modes and mode switching conditions are shown in Fig. 5 and are explained below.
1) Grasp Mode: The system starts in the grasp mode in which the object is fully grasped and a x = a y = 0.The human can pull the object, but as long as the total external force applied to the object (F sum ) is less than the maximum static friction force (F f m a x ), the system stays in the grasp mode.We have F f m a x = μ s F G where μ s is the static friction coefficient between the object and the gripper (effectively, μ s is twice the static friction coefficient between the object and each finger of the gripper).
In the grasp mode we have 2) Slipping Mode: Once F sum exceeds F f m a x , the system switches to the slipping mode in which the object slips between the fingers of the gripper.In the slipping mode, the system equations become where μ k is the kinetic friction coefficient between the object and the gripper (twice the kinetic friction coefficient between the object and each finger of the gripper).It is assumed that the object moves in a straight line, so the object's acceleration is in the direction of motion.While most of the proposed handover controllers in the literature have only two modes of operation (complete grasp and complete release) the slipping mode is essential to our controller.In this mode, the object is allowed to move but it is not completely released.Therefore, the object's downward acceleration can be measured and the system can distinguish between an unwanted collision and a force applied by the user.One of our main contributions in this study is to design a controller that keeps the system in the slipping mode in order to achieve a smooth and safe handover.
3) Release Mode: Once the output of the controller (F G ) becomes zero, the friction force also becomes zero (F f = 0) and the object is released from the gripper.

B. Controller Design
Here, we discuss the general design requirements for a handover controller that works based on the model presented in Section V-A.Later in Section VI, an instance of such a controller is shown and its performance is evaluated through simulation.
For a setup similar to Fig. 4, with a two-finger robotic gripper and a force sensor attached to the robot's wrist, the output of the force sensor (F s ) is equal to the friction force between the gripper and the object It is assumed that the force sensor is calibrated so that the weight of the gripper is eliminated.The controller gets F s x , F s y , and a y , as input and returns F G as output.The control loop is shown in Fig. 6.We assume that the dynamics of the gripper is negligible so that the commanded grip force is directly applied to the object.The plant under control consists of the gripper and the object.The design goals for the controller are: first, the robot should not allow the object to fall; and second, the handover should be smooth and feel natural for the human.These goals can be achieved by controlling the conditions at which the system switches between the three modes, i.e., grasp mode, slipping mode, and release mode.Please note that these three modes are different modes of the plant itself, not the controller.The controller can observe the current mode of the plant, and it may or may not behave differently in each mode.
We next discuss in detail the control requirements at mode switches.
1) Switching From the Grasp Mode to the Slipping Mode: Initially, the system is at rest (F p = 0, a y = 0, F s y = W , F s x = 0).In order to sustain the object, we should have Equation ( 4) can be satisfied by assigning F G initially to be equal to or greater than W μ s .Furthermore, the force overshoot of the system is determined by the conditions at which the system switches to the slipping mode.Force overshoot is defined as the minimum amount of pulling force that the user has to apply before the object is released.Here, we define the start force threshold (F ST (ϕ)) as the minimum amount of pulling force that the user has to apply in a specific direction (ϕ) so that the system switches from the grasp mode to the slipping mode.It is shown later that the force overshoot is equal to F ST .The switching condition (shown in Fig. 5) can be written as In the equation above, F G is to be determined by the controller.F ST can be then computed as a function of ϕ by solving (5) for a given controller.From the practical point of view, it is not possible to achieve a zero force overshoot, as choosing small values for F ST results in a very sensitive system that responds to the measured force noise.
2) Switching Back From the Slipping Mode to the Grasp Mode: Once the system switches to the slipping mode, the desired behavior is not to switch back to the grasp mode unless the pulling force falls below F ST .Otherwise, the system would exhibit chattering.We define stable force threshold F BT (ϕ) as the minimum amount of pulling force that the user has to apply to the object in a specific direction (ϕ) when the system is in the slipping mode so the system does not switch back to the grasp mode.F BT can be computed as follows: Please note that F G is to be determined by the controller and it might have different values in ( 6) and ( 5), since the system is in two different modes.In order to avoid chattering, we should have 3) Switching From the Slipping Mode to the Release Mode: According to Fig. 5, the system switches from the slipping mode to the release mode once F G becomes zero.The controller should be designed so that F G converges to zero for a constant pulling force that caused the system to switch from the grasp mode to the slipping mode.This ensures that no larger force than F ST is required to take the object from the robot and F ST is indeed the system force overshoot.
4) Switching Back From the Release Mode to the Slipping Mode: For a small amount of time after the system switches to the release mode, it needs to keep monitoring the object so that it can react in case the object is falling.We call this period of time the monitoring time (t m ).We should make sure that during t m , the system will switch back from the release mode to the slipping mode if the object's downward acceleration is more than a threshold, namely a y m a x .In the release mode, assuming that the object is taken by the user, we have In case we have a y > a y m a x , The controller should re-establish the grasp by increasing F G .

VI. SIMULATION RESULTS
MATLAB simulations are carried out to evaluate the performance of the controller framework proposed in Section V-B.An instance of a controller designed based on the procedure explained in Section V-B is proposed in our previous work [11], called "fail-safe" (FS) handover controller.We use this controller in our simulations.For the readers' convenience, a review of the FS controller and its parameter tuning procedure is presented in Appendix A. Furthermore, a detailed stability analysis of FS controller can be found in Appendix B. The controller has the same governing control law for all the three modes of the system where n is the time step and α, β, γ, F margin , and k i are constants that should be selected based on the object weight, desired force overshoot, etc. Equation ( 9) is composed of several terms: the term αF s y [n] counteracts the load force, so a heavier object results in a larger grip force.The term βa y [n] balances the object downward acceleration, so the object is regrasped if it is falling.The term −γF s x [n] reacts to the horizontal pulling force.It will be shown in Appendix A that it allows the user to take the object (pull) at a wider range of angles ϕ.The term −F margin allows the system to tolerate small values of the object downward acceleration before regrasping the object.Finally, the term (1 − k i )F G [n] works as an integrator which improves the stability of the system.For details on the design of this controller, how different parameters can be chosen and the stability analysis, refer to Appendices A and B, and [11].
We define two different actions: dropping and taking, similar to what we defined in Section IV.In dropping, a force is applied to the object, but the weight of the object is not compensated, resulting in the object falling if released by the robot.This is to test whether the robot can prevent a failure.In contrast, in taking, it is assumed that the object is taken by a human, so the weight of the object is compensated.In this case, we want to see whether the controller can achieve a smooth handover.The two actions are simulated similarly: in each time step, the pulling force in the desired direction is applied to the object in addition to F G and W .The acceleration of the object is then calculated considering the friction force between the gripper and the object.The only difference between the two actions is that in a taking scenario, the object's weight is canceled out during the transfer phase.Fig. 7 shows the object's weight (object's load) as it is seen by the controller in a taking scenario.In the simulations, we selected the object's mass M = 0.2 Kg and μ s = 0.6, μ k = 0.5.The controller parameters are selected as follows (refer to Appendix A for details): the maximum object's downward acceleration that the system tolerates is set to 4(m/s 2 ).Based on (17), β is set to 3 and F margin is set to 12. Parameter α is then selected equal to 10 to satisfy (15).Also, based on Fig. 18, γ is set to 5 to achieve F ST (ϕ = 0) < 1N, i.e., force overshoot of less than 1 N when pulling the object horizontally.The system parameters used in the simulations are summarized in Table III.
In the first simulation, a horizontal pulling force is applied to the object with ϕ = 0 • .Both the dropping and the taking actions are performed.F p starts from zero at T 0 = 1 s and reaches 2 N at T 1 = 1.5 s.In the dropping scenario, the pulling force becomes zero again at T 2 = 2.5 s.In Fig. 8  in the figure indicate the mode of the system: red corresponds to the grasp mode, yellow to the slipping mode, and green to the release mode.
As shown in Fig. 8(a), in case of taking, F G drops to zero once the pulling force is applied, which means the object is released immediately.Also it is shown in Fig. 8(b) and (d) that while the pulling force is applied, F G is regulated and a y is kept below 4 m/s 2 .Furthermore, once applying the pulling force stops in Fig. 8(b), F G rises back to the initial value, the grasp is re-established and the object's fall is prevented.Another observation is that in Fig. 8(c), it can be seen that the object briefly accelerates downward before it is completely released.The reason is that we intentionally added a slight delay to the weight compensation algorithm to make sure the controller can handle a delay in load compensation.
In the next run, the taking action is simulated with two different angles of pulling.In Fig. 9(a) and (b), the applied pulling force along with the regulated grip force is shown for taking action with ϕ = 20 • and ϕ = −20 • , respectively.The corresponding object accelerations can be found in Fig. 9(c) and (d).In both cases, the object is released immediately; note that in the case of ϕ = −20 • , the object has negative downward acceleration after being released because it is pulled upward.
In order to investigate the effect of the friction coefficient on the behavior of the system, the simulation is repeated with ϕ = 0 • , this time with two different friction coefficients.In Fig. 10(a) and (b), the object acceleration is shown for the taking and the dropping actions for a surface with high friction (μ s = 0.9, μ k = 0.8).In Fig. 10(c) and (d), the object acceleration is shown for the same actions for a surface with low friction (μ s = 0.3, μ k = 0.2).In both of the taking cases, the object is released immediately after being pulled as expected.In the dropping cases, the object is not released and the grasp is re-established after the pulling force is removed, although the object's slipping acceleration is higher in the system with low friction.
The next simulation investigates the sensitivity of the system.In Fig. 11(a) and (c), the forces applied to the object and the  object's acceleration are shown for the dropping action under the presence of a white Gaussian noise (SNR = 3 dB) applied to F G .It can be seen that the object's downward acceleration is still maintained below 6 m/s 2 and the grasp is re-established after the pulling is stopped.In Fig. 11(b) and (d), the same signals are shown in a dropping action while there is a white Gaussian noise (SNR = 3 dB) applied to a y .The figure clearly shows that the system is able to prevent the object from falling and re-establishes the grasp after the pulling is stopped.This shows that the overall performance of the system is quite robust against the sensor noise and the gripper disturbance.

VII. IMPLEMENTATION AND EXPERIMENTS
The FS controller is implemented on a Baxter robot [33] with a parallel gripper [34] installed on one of the robots' arms.In order to control the grip force, we used an open-loop positionbased method: a small metal plane is added to one of the fingers of the gripper with a linear spring between the plane and the finger.Therefore, the grip force can be controlled based on the gripper position where k is the stiffness of the spring, x is the gripper position command, and x 0 is the position at which the gripper touches the object.The stiffness of the spring (k) was experimentally determined to be 5 (N/cm).The gripper position control loop operates at 25 Hz.
In order to measure the force, an SI-65-5 ATI Gamma force sensor [35] is added to the robot's arm between the end effector and the gripper.The force sensor output is sampled at 100 Hz and the data are transferred to a computer through a PCI-6034 NI data acquisition board [36].A 6 cm × 7 cm × 17 cm empty box is used as the test object.The object weight is 230 ± 5 mg.The object downward acceleration is measured through an ADNS-2051 optical sensor [32] installed on one of the fingers of the gripper.The object, the gripper, and the installed devices on the gripper are shown in Fig. 12.

A. Experiment One: Evaluating the Smoothness of the Controller
In order to evaluate the smoothness of our FS controller and to fine-tune the parameters of the controller, we designed a robot-to-human handover experiment.Baxter would hand the object over to human subjects, using different controllers.In each trial, Baxter grasps the object within its electrical gripper.The handover controller is then activated and a beep sound is played for the subject.Participants are told to take the object from the robot, after hearing the beep sound.
1) Handover Controllers: We began with implementing the fail-safe handover controller with the same parameters used in the simulations (see Table III).Prior to experiment one, we conducted a pilot study with four participants (in addition to the subjects in experiment one), to see if the controller can perform the handover task smoothly without dropping the object.We realized that the system is very sensitive to measured force noise and in several cases, it dropped the object before the participants touched it.Therefore, we decided to increase the force overshoot by decreasing parameter γ, from 5 to 3. Also we increased F margin from 12 to 15 N to increase the maximum tolerated object's downward acceleration.We used the following four different handover controllers in experiment one.
1) First fail-safe handover controller (FS1) with parameters α = 10, β = 3, γ = 3, F margin = 15, k i = 0.13.This controller should have a performance similar to the controller used in the simulations.According to Fig. 18, with having γ = 3, the force overshoot is about 2 N when pulling forward (F ST (ϕ = 0) 2 N). 2) Second fail-safe handover controller (FS2) with parameters similar to FS1, except that α was increased to 15.This controller is used in the experiment to investigate the effect of changing parameter α.According to (18), this controller has force overshoot about 4 N when pulling forward.3) Third fail-safe handover controller (FS3) with parameters similar to the FS1 controller, except that γ was decreased to 1.This controller is used in the experiment to investigate the effect of changing parameter γ.The force overshoot is about 3 N when pulling forward.4) The fourth controller is HI handover controller [9].The HI controller was proposed in [9] and it was shown to have a smooth performance compared to the other existing handover controllers.We thus wanted to compare the smoothness of our FS controller to that of the HI controller.The HI controller regulates the grip force according to the object's load force, in a linear fashion where F G is the applied grip force, F L is the object's gravitational load force acting on the robot's gripper, m is a constant slope, and F ZLG is a nonzero amount of grip force applied at zero load force that acts as a safety margin [9].The HI controller output is illustrated in Fig. 13.In the figure, F Lo is the total weight of the object supported by the robot at the beginning of the handover when the robot has stably grasped the object.At this time, the robot applies an initial grip force of F Go that has been properly calculated so the object does not slip.The object is completely released only after a slight upward pulling force of ε is sensed.The slope parameter (m) can be calculated based on F Lo , F Go , and F ZLG .We selected the controller parameters so that it mimics the behavior of Controller A introduced in [9], which was shown to be preferred by humans: ε was set to −1.0 N (according to [9], ε should be around 50% of the object's weight), F ZLG was set to 2 N to account for the sensor measurement errors and F Go was set to 7 N, equal to the initial grasp force selected for FS1 controller so the comparison would be fair.2) Experimental Procedure: Each participant compares three pairs of controllers: (FS1 versus FS2), (FS1 versus FS3), and (FS1 versus HI).In each comparison, the robot hands over the object two times with one controller, followed by another two times using the other controller.After a set of four trials, the subject is asked to answer a survey evaluating the behavior of the robot in the first two trials compared to the second two trials.To eliminate ordering effects, each controller pair was presented to the participants in both orders.That results in a total of 6 pairwise comparisons and 24 trials.To balance the carryover effects, we used a complete Latin square design [37].
To observe the effect of the pulling direction on the results, the participants are asked to perform the entire procedure twice, once while standing and once while sitting, resulting in a total of 48 trials per participant.We expected that the participants would pull the object in different direction when they are sitting compared to standing situation.
We put a table with dimension (L : 75 cm; W : 60 cm; H : 70 cm) at a distance of 70 cm in front of the robot.The subject is sitting/standing across the table at a distance of 15 cm from the edge of the table.The object is transferred at a height = 25 cm above the table and at a distance = 40 cm from the edge of the table in front of the subject.Fig. 14 shows the position of the subject and the robot in standing and sitting cases.
The survey questions are as follows.1) Rate how easy it was to take the object from the robot in the first two trials.There are five options, very easy, easy, moderate, hard, and very hard.2) Rate how easy it was to take the object from the robot in the second two trials.There are five options, very easy, easy, moderate, hard, and very hard.3) Do you prefer the robot behavior in the first two trials or in the second two trials?There are three options, I prefer the robot behavior in the first two trials, I prefer the robot behavior in the second two trials, and No preference.For the study, 20 participants (13 males and 7 females) between ages 23 to 32 were recruited from students and staff at the UIC, by posting flyers and sending e-mails to UIC graduate students' e-mail list.Before starting the experiment and to get familiar with the robot, each participant performed the handover task eight times while standing, including two times with each controller in a random order.Participants were informed that there is no movement in the robot's arm during the task.No further instructions were provided to the participants about how to take the object.3) Results: We used the sign test [31] to analyze the survey responses.
Participants' ratings for how easy it was to take the object from the robot did not significantly differ between the FS1 controller and the HI controller (Z = −0.99,p < 0.324), or between the FS1 controller and FS3 controller (Z = −1.57,p < 0.12).However, significantly more participants responded that the object can be more easily taken from the robot with the FS1 controller compared to the FS2 controller (Z = −2.65,p < 0.008).
While the system is in the grasp mode, the applied pulling force can be calculated using (1) and (3).The average applied pulling force and the average produced grip force for the standing trials are shown in Fig. 15 for all of the controllers.The force signals are normalized over time (t = 1 shows the time that the object is released).Force overshoot for each controller can be determined from Fig. 15(a) by measuring the maximum pulling force applied before the object is released.It can be seen in Fig. 15(a) that the FS1 controller force overshoot (∼2 N) is smaller compared to other controllers.b) Sitting case: We speculated that in the sitting case, a subject would pull the object almost horizontally as the object is in front of his/her upper body.Since the HI controller can only handle vertical object transfer, we predicted the force overshoot for this controller to be significantly higher when the user tries to take the object horizontally.We also predicted larger force overshoot for the FS2 and FS3 controllers in this case, since F ST (ϕ = 0) is larger for these controllers compared to the FS1 controller.The average applied pulling force and the average produced grip force for sitting trials are shown in Fig. 16 for all of the controllers.The force signals are normalized over time  (t = 1 shows the time when the object is released).It can be seen in Fig. 16(a) that the FS1 controller force overshoot (∼2 N) is smaller compared to other controllers.In fact in this case, there is a significant difference between the force overshoot of the FS1 controller and that of the other controllers.
The survey analysis shows that in the sitting case, significantly more participants responded that the object can be more easily taken from the robot with the FS1 controller compared to the HI controller (Z = −3.268,p < 0.0011), to the FS2 controller (Z = −4.268,p < 0.0001), and also to the FS3 controller (Z = −2.370,p < 0.018).Also, significantly more participants responded that they prefer the FS1 controller over the HI controller (Z = −3.86,p < 0.0002), over FS2 controller (Z = −2.95,p < 0.004), and also over FS3 controller (Z = −2.65,p < 0.008).The subject preference ratings can be explained by the smaller force overshoot of FS1 controller.
In summary, we conclude that the FS1 controller shows a superior performance compared to the other controllers in terms of the force overshoot (smoothness) and the subject preference ratings.

B. Experiment Two: Evaluating the Object Safety
The second experiment is designed to evaluate the object safety.Following the same method that we used in Section VI, a failure is simulated for the robot: a forward pulling force of specified magnitude is applied to the object without compensating the weight of the object.A rope is attached to the object with an analog force meter attached to the rope.At t = 3 s the rope is pulled with F p = 2 N and ϕ = 0.For this experiment, we use FS1 controller from Section VII-A which showed superior performance compared to the other controllers.
Fig. 17 shows the applied pulling force, the generated grip force, and the object downward acceleration.As it is shown in Fig. 17(b), the object downward acceleration a y is kept below 4 m/s 2 while the pulling force is applied, which means that the object is not released completely and the system has remained in the slipping mode, successfully preventing the object from falling.

VIII. CONCLUSION
We proposed a novel framework for ensuring a safe and smooth robot-to-human handover.The framework critically depends on the ability of the robot to readily detect a failure during the handover and effectively recover from it.This in turn motivated us to study how humans detect failure when they play the role of a giver.Toward this goal, we conducted a human study to investigate which sensing modality is used by humans to detect failure.In particular, we examined whether haptic sensing plays the dominant role in detecting failure, or the human givers primarily depend on vision.The results suggest that at least in some scenarios, haptic sensing is not reliable enough to determine whether the object is dropped or successfully taken by the receiver after being released, and that humans seem to rely on vision.Motivated by this finding, we proposed a robot handover architecture that relies on measuring relative motion between the object and the robot hand, provided through an optical sensor, to detect an impending handover failure.In turn, a handover controller was proposed that uses the measurement of object acceleration in the feedback loop to guarantee handover safety.At the same time, by monitoring the force applied on the object, the controller achieves smoothness of the handover.The proposed architecture thus overcomes the shortcomings of the existing controllers that tradeoff smoothness for safety, or vice versa.
We provided a detailed analysis of the proposed controller.The controller is designed to work in three different modes: grasping, slipping, and the release mode.One of our main conceptual contributions is to explicitly model the slipping mode.The slipping mode characterizes the transfer phase of the handover and is, therefore, critical for both safety and smoothness.By monitoring the object acceleration and applied forces, the controller is able to distinguish between a slipping object that is falling and a slipping object that is being transferred from the robot's hand to the human.Performance of the proposed handover controller was first investigated through simulation and it was shown that the controller is quite robust.Also, we showed experimentally and by assessing human satisfaction that the proposed system demonstrates significantly higher performance compared to other handover controllers.Given that the proposed system is inexpensive and easy to implement on general robot hardware, we believe that it represents a significant step toward improving physical human-robot interaction.

APPENDIX A FAIL-SAFE HANDOVER CONTROLLER
We proposed an FS handover controller in our previous work [11] that is designed based on the procedure explained in Section V-B.The FS controller regulates the grip force for a twofinger robotic gripper with a force sensor attached to the robot's wrist and an optical sensor installed on the gripper that can measure the object's downward acceleration.The system setup and the controller loop are illustrated in Figs. 4 and 6.Here, we present a review of the FS controller.
The controller law is proposed as below where α, β, and γ are constant values.The grip force is chosen based on the vertical load force in a linear fashion.In this way, the grip force decreases as the user compensates the vertical load force.It is shown in [2] that the grip force must indeed decrease monotonically with the vertical load force in order to achieve a smooth handover.Furthermore, in order to prevent the object from falling, the object's downward acceleration is included in the controller equation.A higher downward acceleration of the object will result in a larger grip force.Also, we want the human to be able to take the object not only vertically, but in any direction.Therefore, the x component of the measured force (F s x ) is also included by a negative factor so as F s x increases, the grip force will be decreased.Please note that F G is bounded below by zero (F G m in = 0).According to (12), after the object is released from the gripper (F s x = F s y = 0), even a small value of the object's downward acceleration results in the grip force becoming nonzero, so the system will switch back from the release mode to the slipping mode.In order to solve this issue, a term −F margin is added to the controller that makes the system tolerate a small downward acceleration: It is shown in Appendix B, that F margin helps the stability of the system when the system is in the release mode.In discrete time, the controller equation becomes The coefficients α, β, and γ in ( 14) should be selected properly to regulate the mode switching conditions, as described in Section V-B.
1) Rest Condition: In the grasp mode, when the system is at rest (F p = 0, a y = 0, F s y = W, F s x = 0), in order to satisfy (4), we should have The above is obtained by substituting F G from ( 13) into (4).α should satisfy (15) so the object does not fall in the grasp mode.
2) Regrasp Condition: In the release mode, we have F s x = F s y = 0. Therefore During the monitoring time (t m ), the maximum object's downward acceleration that the system tolerates is In case we have a y > a y m a x , the system will re-establish the grasp by switching back to the slipping mode.
3) Force Overshoot: To calculate F ST , we consider the switching condition from the grasp mode to the slipping mode at the extreme: F sum = μ s F G .Expanding F sum and substituting F G from (13) into (5) we have Equation ( 18) establishes a relation between F ST and the direction of pulling.In Fig. 18, F ST is shown with respect to the pulling angle ϕ for a specific set of parameters specified in Table III and three different values of γ.According to Fig. 18, for a specific value of γ, there is a maximum angle (ϕ max ) at which the user can successfully take the object from the robot by pulling it in that direction.For γ = 2.5, we have ϕ max = 55 • .Higher values of γ result in lower start force threshold and wider range of angles at which the user can successfully take the object.
Besides the procedure explained here, the controller parameters can be selected using online learning schemes such as the one proposed in [38].

APPENDIX B STABILITY ANALYSIS FOR THE FAIL-SAFE HANDOVER CONTROLLER
Stability of robots physically interacting with humans has been extensively studied [39]- [45].The primary focus of these investigations are robots that are physically coupled with humans such as assistive devices and haptic interfaces, where stability is necessary for safety.The common approach in these studies is to model the human (and the environment if needed), either explicitly as an impedance or admittance, or implicitly as a passive subsystem, and the stability of the overall system is then examined.
In tasks such as a handover, it is more accurate to characterize the actions of the human as the exogenous inputs for the robot; the robot needs to generate an appropriate action in response.As a result, the issue of stability reduces to stability of the robot controllers implementing specific robot actions for arbitrary input.
In the case of the handover, the primary response of the robot is the applied grip force F G .We will assume that the dynamics of the gripper is negligible so that the commanded grip force is directly generated by the robot.The stable behavior of the robot thus reduces to two separate conditions: (a) for a constant pulling force F p (human input), the grip force F G should converge to a constant; and (b) for a constant pulling force F p , the system should not switch between different modes (grasp, slipping, and release).
Assuming the system is in the grasp mode, we can determine F G by substituting (1) and ( 3) into ( 13) This shows that in the grasp mode, condition (a) is satisfied.
To prevent consecutive mode switches while F p is constant, the stable force threshold (F BT ) should be smaller than the start force threshold (F ST ) as stated in (7), so the system does not switch back to the grasp mode once it switched to the slipping mode.To calculate F BT , we find sin θ and cos θ from (2) and substitute them into (3).F s x , F s y , and a y can then be found in terms of F G , F p , and ϕ Substituting ( 20) into ( 14) we have For (21) to be stable and produce bounded output, we should have |a| ≤ 1 (24) so the single pole of the system (z = a) would be inside the unit circle and the closed-loop system becomes stable.Please note that the dynamics of the gripper is neglected.In the extreme case when |a| = 1, F p would be equal to F BT .This gives us F BT as a function of the direction of pulling.In Fig. 19(a), F BT is shown with respect to pulling angle (ϕ) for a set of system parameters specified in Table III.As it is shown in Fig. 19(a), F BT is greater than F ST .In fact, it can be shown that the minimum required force to have a stable system is greater than the start force threshold for any set of system parameters.As a result, when applying an ascending pulling force, the system will switch to the slipping mode once F p exceeds F ST and then it will switch back to the grasp mode before F p reaches F BT .This is an unstable situation.
To resolve this issue, we add an integrator term to the system Therefore, (22) becomes Again, we put |a| = 1 to calculate F BT .In Fig. 19(b), F BT and F ST are shown as a function of the pulling direction.Fig. 19(b) shows that the integrator term has pushed the pole of the system inside the unit circle so the system has become stable for all values of the pulling force.It can be shown with further analysis that adding the integrator term not only satisfies (|a| ≤ 1), but also results in F G to be descending for a constant pulling force F p and, therefore, guarantees the system to stay in the slipping mode.

Fig. 1 .
Fig. 1.Simulation of the object transfer phase: the object load is gradually transferred from the giver to the receiver during t tran sfer .(a) Giver load force (F L G ).(b) Receiver load force (F L R ).The simulation mimics the forces shown in the first panel in [2, Figs. 4 and 5].

Fig. 3 .
Fig. 3. Pulling action in (a) taking horizontally, (b) dropping horizontally, (c) taking diagonally, and (d) dropping diagonally cases.The blue arrow shows the horizontal pulling direction.The red arrow shows the diagonal pulling direction.

Fig. 4 .
Fig. 4. Two-finger gripper and applied forces to the object.

Fig. 7 .
Fig. 7. Object's load as seen by the controller in a taking scenario.
(a) and (b), the applied pulling force and the controller output are shown for the taking and dropping actions, respectively.The acceleration of the object for both actions is shown in Fig. 8(c) and (d).The background colors

Fig. 10 .
Fig. 10.Acceleration of the object in (a) taking with high friction, (b) dropping with high friction, (c) taking with low friction, and (d) dropping with low friction grippers.Red background indicates the grasp mode, yellow the slipping mode, and green the release mode.

Fig. 11 .
Fig. 11.Applied pulling force and grip force in dropping with (a) disturbance applied to F G and (b) noise applied to a y .Acceleration of the object in dropping with (c) disturbance applied to F G and (d) noise applied to a y .Red background indicates the grasp mode, yellow the slipping mode, and green the release mode.

Fig. 15 .
Fig. 15.(a) Average applied pulling force and (b) average grip force for all of the controllers-standing trials.Force signals are normalized over time.

Fig. 16 .
Fig. 16.(a) Average applied pulling force and (b) average grip force for all of the controllers-sitting trials.Force signals are normalized over time.

Fig. 17
Fig. 17.Problematic handover: (a) applied pulling force and the grip force; (b) object downward acceleration a y .
Fig. 17.Problematic handover: (a) applied pulling force and the grip force; (b) object downward acceleration a y .

Fig. 18 .
Fig. 18.F S T versus the pulling angle ϕ, plotted for three different values of γ.

Fig. 19 .
Fig. 19.F S T and F B T with respect to pulling angle ϕ in (a) the primary controller design and (b) after adding integrator term.

TABLE I OBJECT
ACCELERATION IN THE DIRECTION OF PULLING TABLE II PERFORMANCE OF THE PARTICIPANTS Participants were instructed not to regrasp the object if they felt it was dropped.