A Preliminary Study of Peer-to-Peer Human-Robot Interaction

The Peer-To-Peer Human-Robot Interaction (P2P-HRI) project is developing techniques to improve task coordination and collaboration between human and robot partners. Our work is motivated by the need to develop effective human-robot teams for space mission operations. A central element of our approach is creating dialogue and interaction tools that enable humans and robots to flexibly support one another. In order to understand how this approach can influence task performance, we recently conducted a series of tests simulating a lunar construction task with a human-robot team. In this paper, we describe the tests performed, discuss our initial results, and analyze the effect of intervention on task performance.


I. INTRODUCTION A. The Peer-to-Peer Human-Robot Interaction Project
Since February 2004, NASA has been pursuing the "Vision for Space Exploration" (WE) [I].The primary goal of the VSE is to "implement a sustained and affordable human and robotic program to explore the solar system and beyond", beginning with robotic missions to the Moon in 2008 and leading eventually to human exploration of Mars.To achieve this goal, humans and robots will need to collaborate to perform basic mission tasks including structural assembly, inspection, and maintenance.
The objective of the "Peer-to-Peer Human-Robot Interaction" (P2P-HRI) project, therefore, is to develop HRI techniques that allow humans and robots to work effectively together [2].We use the term "peer-to-peer" not because we expect humans and robots to have equal capabilities, but to emphasize the idea that humans and robots should work as partners, Le., that their roles should be as balanced as possible.In particular, we believe that it is possible to compensate for failures and limitations of autonomy through interaction and teamwork.
Our research focuses on making human-robot teams more effective for space exploration by: (1) developing natural interaction mechanisms; (2) reducing the workload associated with robot control; and (3) maximizing the work that the humans and robots can accomplish together.In addition, we are developing methods that enable human-robot teams to operate independently of ground control.This capability is ration in deep craters) or significantly time-delayed.
There are three primary components in our approach.First, we are developing an interaction framework called the "Human-Robot Interaction Operating System" ( W O S ) .The W O S is designed to enable humans and robots to engage in task-oriented dialogue and problem solving.Second, we are using computational cognitive architectures to model human behavior and make human and robot more understandable to each other.Finally, we are developing evaluation procedures using human-robot teams, analog environments, and quantitative HRI metrics [3 1.

B. Human-Robot Interaction Operating System
In order for humans and robots to work effectively together, they need to be able to converse about abilities, goals and achievements.The Human-Robot Interaction Operating System (HRUOS) is an agent-based system that provides coordination and interaction services for human-robot teams [4].A key feature of the W O S is that it allows robots to ask questions of humans, so that robots can obtain human assistance for cognition and perception tasks.
We designed the W O S to support the performance of operational tasks.In space exploration, operational tasks include: construction, structural inspection and maintenance, and in-situ resource collection and transport [5].The current version of the HWOS is implemented using the Open Agent Architecture ( O M ) [6] and the Internet Communication Engine (ICE) middleware [7].

neededlcommunicationslinterminent(e.g.,explo-
The primary components of the HRUOS (Figure 1) are: I ) Task Manager: The Task Manager (TM) is responsible for coordinating and managing the execution of operational tasks.It does this by decomposing the overall goal of the system into high-level tasks, which are assigned to humans or robots for execution.Only a single task is assigned to a given agent at a time.Unlike traditional executives, the TM does not know anything about low-level task details.Instead, it relies on each agent to work in a distributed, independent manner, managing and monitoring their own task execution.
2)  a task to be performed or a question to be answered, the RM generates a prioritized list of agents to be consulted.Unlike facilitation in most agent systems, the RM performs delegation using multiple criteria that can vary with time and situation, rather than simple match making.In particular, in addition to service matching, the RM considers a variety of factors including availability, physical location, workload, past performance, etc.
3) Interaction Manager: The Interaction Manager (IM) coordinates dialogue between agents.The IM gives each agent the ability to communicate with other agents: to ask for help, to respond to questions, etc.In the HRVOS, agents communicate with one another via point-to-point "dialogue endpoints".Endpoints allow agents to send a variety of data (text, images, sensor readings, etc.) to each other.The W O S currently includes graphical and speech user interfaces.Other interface modalities (e.g., visual gesturing) will be included in the future.

4) Context Manager:
In a complex agent system, keeping track of the activities and state of agents over time is a difficult task.This is paaicularly true when multiple agents operate in parallel and when activity is observed remotely (e.g., via user interfaces).Thus, to facilitate situational awareness, we have developed a Context Manager (CM).The CM keeps track of everything that occurs while the system is running: task status and execution, agent activities, agent dialogue, etc.Then, when agents have need to recall history, they can query the CM for a summary of information.

5) Spatial Reasoning Agent:
When human-robot teams perfom operational tasks, such as construction, understanding and communicating spatial dialogue plays a significant role [e]-[ 101.In particular, when humans and robots work in a shared environment, robots must be able to understand how humans perceives space and the relative positions of objects around them.Thus, the HRVOS includes a spatial reasoning agent (SRA), which resolves spatial ambiguities in dialogue (e.g., "move the light to the left of the box").The current implementation of the SRA is capable of resolving frame of reference ambiguities including ego-, addresse-, object-, and exo-centric references using a computational cognitive model

A. Use Case Study
In November 2005, to assess our progress in developing with peer-to-peer HRI, we conducted a study of a simulated construction project involving seam welding and inspection by a human-robot team.Seam welding is a task that will be required for building and maintaining a variety of structures on planetary surfaces [14].For example, linear welds might be used to construct pressure vessels, work hangers, and shelters too large to raise into space in one piece [15].
The human-robot team consisted of five different members working in parallel (Figure 2).In each trial, the team worked to retrieve panels from a storage depot, place the panels on a structural frame, weld adjacent panels, and inspect the welds.Two people wearing spacesuits performed the roles

B. Data Collection and Baseline
During the study, we made extensive use of video and audio to record human-robot interaction.Three observers used logging software to code human and robot activity.For each session, we logged time spent moving, time on task, and intervention time (the time a human or robot spent helping another agent).From this data we are able to determine the free time that any of the agents had.
To establish baselines, we collected the time required for each agent to perform primitive tasks: retrieve and assemble panels (astronauts only), move between panels (all), seam weld (astronaut and Robonaut), inspect weld (K10 only).
Table I shows the baseline data for the EVA astronauts, K10, _and_Rabanaut~s~as_eline_data_canhe_led to construcL time-based workfiows of different team configurations.  of astronauts in EVA', retrieving and placing the panels.The Robonaut humanoid robot [16] from Johnson Space Center worked as a welder using a simulated seam weld process.The K10 rover from Ames Research Center was responsible for inspecting the welded seams.An additional person, WA2, remotely supported the construction project from inside a habitat mockup.
This work scenario provides numerous opportunities for dynamic and flexible human-robot interaction.For example, a variety of communication acts are possible: human generated commands, questions from the robots to the human, etc.Additionally, humans may remotely interact with the robots (e.g., remotely deploy the inspection robot from a habitat) as well as working side-by-side with others (e.g., leading the welder robot to a site ready for welding).We should note that it is unlikely that human-robot teams will be able to work as quickly as comparably sized human teams any time soon.But, beqause cost pressures and other constraints (e.g.payload launch capacity) will keep astronaut teams small, exploration missions will need to include teams of humans and robots.Thus, we are most concerned with identifying task workflows that maximize human-robot team productivity, while reducing the number and duration of EVA'S.

C. Preliminary Results
One series of tests included seven trials, in which six panels were placed, welded, and inspected by the humanrobot team.The start condition and work sequence in each trial was identical.During these trials, the two EVA astronauts commanded and provided assistance to the robots.The N A astronaut communicated with the EVA astronauts.N A also teleoperated and assisted K10 with inspections when needed.Trials 2 to 7 were successfully completed.Figure 3 shows a timeline from Trial 6, broken down into four activity categories: free time, time on task, time on task with help, and intervention (providing help).
To better understand how the EVA astronauts worked, we computed the percentage of time the astronauts dedicated to each activity category during each trial (Figure 4).The astronauts spent about the same percentage of time on the task during all runs.They spent little time intervening (providing help to robots).Overall, the EVAs had a high percentage of free time, most of which occurred while waiting for robots to finish working at the end of each trial and which would be available for other work.For the N A we coded four work tasks: monitoring (user interface or line-of-site observation), robot teleoperation; intervention (providing help to robots), and human communication.During all trials, we also asked the IVA to perform a secondary task (reading a newspaper) to better simulate actual work (i.e., so that his attention would not be focused only on the team).Figure 5 shows the percentage of time the IVA spent in each of these activities.By far the greatest percentage of the IVA's time was spent on communication.This reflects a need for having the status of the robots and TM more visible to the EVAs.In Trial 7, the IVA was teleoperating K10 while communicating with EVAs but this was not the norm.

INTERVENTION AND EVA
To better understand how human intervention (Le., an astronaut assisting a robot) effects EVA, consider the following two measures of EVA productivity: 1) The amount of useful work that astronauts can do.
2) The amount of time astronauts must spend in EVA.Depending on the nature and the context of the task, it may be important to maximize the first measure (useful work).For other tasks, minimizing the second (exposure) may be the priority.The ideal, of course, is a system that can maximize useful work while minimizing exposure.
Interventions reduce the amount of time the astronauts have available for other work in EVA.Because of the time associated with egresg and regress, if robots are ''unreliable'' and require many interventions, the astronauts may need to stay in EVA and "wait" to handle for these unplanned interventions.Even if the astronauts were to remain out on EVA trying to get other work done, their work would be constantly interrupted by robots requesting intervention.This would directly reduce the productivity of the human-robot team.
Figure 6 illustrates the effects of interventions on different parts of an EVA timeline.A nominal timeline in which the human-robot system performs a collaborative task without unplanned interventions is shown in Figure 6a.The maximum EVA time, TEVA, is the maximum amount of time that the human-robot system has to perform a primary task (Task 1).The nominal amount of time for the human-robot system to perform a specific task is labeled T t a s k .The nominal amount of time required for the astronauts to perform their part of Task 1 is labeled TtaskEvA.The time remaining once the astronauts complete Task 1 is the time available for them to perform other work.
Interventions that occur while the astronauts are still performing their part of Task 1 increase TtaskEva and T t a s k as shown in Figure 6b.Unplanned interventions after TtaskEVA lead to a situation in which the astronauts may have to remain in EVA in order to intervene.If the astronauts do not remain in EVA, time penalties associated with egresshegress may be incurred.This situation is shown in Figure 66.Each of these events significantly increase exposures and decreases the time available for other work.
If, however, the human-robot system is very "reliable" (i.e., robots do not often need help), then astronauts have two choices.On the one hand, when they finish their primary task, they can stay in EVA and begin working on other tasks.This increases the amount of useful work.Alternatively, the astronauts can assume that the robots will complete their tasks without intervention, and thus they can leave EVA.This helps minimize exposure.In other words, a very "reliable" human-robot system provides the option to maximize different productivity measures.
We can examine the effect of intervention on EVA using an analysis inspired by classic reliability techniques [17].To begin, we first define the following terms: a Mean Time Between Znterventions (MTBZ) is the mean time that a human-robot system operates nominally (i.e., outside of intervention).MTBI is analogous to Mean Time Between Failures (MTBF) [18].Mean Time to Intervene (MTTZ) is the mean time humans and robots spend in intervention.MTTI is analogous to Mean Time To Repair (MTTR) [ 181.MTBI is indicative of how often intervention is needed (e.g., how frequently robots ask for help).MTBI is a function of several factors: the environment (and uncertainty in the environment) the system or robot is operating in; the autonomy of the system or robot; and inherent component or system reliabiliries.
MTTI is indicative of intervention time (e.g., how long it takes to answer a robot's question).MTTI is a function of many variables, including (but not limited to): the nature of the problem requiring intervention; the amount of information a robot or astronaut is able to gather about the problem; the amount and type of information that can be transferred from robot to human; the physical distance between human and robot; the lag in communications between human and robot; and available resources.

A. Time Available for Other Work
The lifetime of a system is usually considered as having three phases: (1) infant mortality with decreasing failure rate, (2) normal life and (3) wear-out with an increasing failure rate.During normal life, the conditional probability of failure, A, given that a component has survived to a given time t is generally constant.To examine the influence of MTBI and MTTI on EVA, we will focus on the normal life phase.Assuming that X is constant (i.e. that interventions will occur at a constant rate): 1 X we can compute the time available for other work using: where Tother is the "time for other work", TEVA is the total EVA time, TtaskEvA is the nominal time required for astronauts to perform their part of the task, and T t a s k is the nominal time for the human-robot team to complete the task Figure 7 shows how time for other work depends on MTBI, assuming constant MTTI and representative EVA values.Increases in MTTI reduce the time available for human agents to do other work.However as MTBI is increased, the sensitivity to MTTI quickly decreases.Also, increases in MTBI result in increased time available for other work, and the sensitivity to MTBI increases as MTTI increases.This suggests that a designer may be able to compensate for large, or uncertain, MTTI and increase time available to do other work by modestly increasing MTBI.

B. Intervention Probability
To examine how MTl3I and MTTI influence the probability that astronauts will need to intervene, we model the occurrence of interventions using a Poisson distribution.The probability, F(t), that at least one intervention will occur between when the astronauts finish their part of the task and when the human-robot system finishes the task is then:

T t a s k E V A , t o t
where TtaSkEvA,tOt is the time required for the astronauts to perform their part of the task (including interventions) and Ttask,tot is the total t i m e to perform the task.
Figure 8 shows the probability that intervention is required (after the astronauts have finished their part of the primary together, supporting one another regardless of spatial distribution, communication channel, and user interface. During the next few years, our goal will be to apply peer-to-peer HRT.directly to a variety of exploration mission systems, including the Crew Exploration Vehicle (CEV) and lunar surface systems (landers and rovers).In addition, we plan to study how peer-to-peer HRI can be applied to a wide range of in-space and surface exploration activities including structural assembly, inspection, engineering survey and resource mapping, payload transport, site preparation, and infrastructure deployment.

ACKNOWLEDGMENT
We would like to thank Bill Bluethmann, Dan Christian, Lany~waras,JonnHu,ravltnraKaJ~an,cii;-'jrbung, task) as a function of MTBI, assuming constant MTTI and representative EVA values.Increasing MTTI by an order of magnitude only shifts the curve slightly.Furthermore, as MTBI increases, the effect of changes in MTTI or MTBI decreases.This suggests that uncertain MTTI may not significantly influence the probability of intervention.

IV. FUTURE WORK
The use case study revealed a number of deficiencies with the current system.In particular, there were a number of times when astronauts used incorrect dialogue.Because the current speech interface does not provide confirmation, the astronauts did not know if a command was received or if the grammar was incorrect.Thus, both the dialogue and the speech interface need to be made more flexible and robust.
We also observed that the astronauts sometimes forgot to inform the TM when they had completed their work.Although the TM tracks the progress of robot agents, a similar facility is needed for tracking humans in their work.This will require more extensive cognitive modeling and the use of activity monitoring techniques.In addition, our longterm goal is to extend the HRVOS to support large humanrobot teams.To do this, the TM will need to reason about, and plan for, resource usage.
By far the biggest issue, however, is that of status monitoring.At present, it is difficult for EVA astronauts to assess the status of robots (progress, health, etc.) and software agents.This deficiency impacts both situation awareness and context switching.Although it is possible for the IVA astronaut to provide status reports as an overseer, frequent human-human communications make it difficult for the IVA to accomplish other tasks.Thus, we need to develop status monitoring mechanisms that can function independent of the IVA.

V. CONCLUSION
The tools and techniques that the Peer-to-Peer Human-Robot Interaction project is developing have significant potential for improving the human-robot teams.In particular, we believe that software frameworks such as the HRVOS will enable humans and robots to work more productively the k e s K10 team and the JSC Robonkt group for their tireless efforts and contributions to this project.This work was sponsored by a grant (HRT-ICP-04-0000-0155) from the NASA Exploration Systems Mission Directorate.

Fig. 2 .
Fig. 2.Human-robot team working on a simulated construction project.During the study, two astronauts placed panels on a frame, the Robonaut humanoid robot created seam welds, and the K10 mobile robot inspected the welds.A third person (not shown) remotely supported the team from inside a habitat mock-up.

'
Extravehicular Activity (EVA) is work performed by an astronaut outside 21ntravebicular Activity (IVA) is work performed inside a pressurized of a spacecraft.module (spacecraft, habitat, etc.

Fig. 8 .
Fig. 8. Probability of Intervention as a function of MTBI and MITI.EVA values used: Y ~V A = x hours, ykaskEVA = 5=1iask Resource Manager: The Resource Manager (RM) is responsible for continually tracking which resources (human, robots, services) are available and what capabilities each resource possesses.When an agent has a request, such as