Autonomous Learning of High-Level States and Actions in Continuous Environments

How can an agent bootstrap up from a low-level representation to autonomously learn high-level states and actions using only domain-general knowledge? In this paper, we assume that the learning agent has a set of continuous variables describing the environment. There exist methods for learning models of the environment, and there also exist methods for planning. However, for autonomous learning, these methods have been used almost exclusively in discrete environments. We propose attacking the problem of learning high-level states and actions in continuous environments by using a qualitative representation to bridge the gap between continuous and discrete variable representations. In this approach, the agent begins with a broad discretization and initially can only tell if the value of each variable is increasing, decreasing, or remaining steady. The agent then simultaneously learns a qualitative representation (discretization) and a set of predictive models of the environment. These models are converted into plans to perform actions. The agent then uses those learned actions to explore the environment. The method is evaluated using a simulated robot with realistic physics. The robot is sitting at a table that contains a block and other distractor objects that are out of reach. The agent autonomously explores the environment without being given a task. After learning, the agent is given various tasks to determine if it learned the necessary states and actions to complete them. The results show that the agent was able to use this method to autonomously learn to perform the tasks.


I. INTRODUCTION
W E would like to build intelligent agents that can autonomously learn to predict and control the environment using only domain-general knowledge.Such agents could simply be placed in an environment, and they would learn it.After they had learned the environment, the agents could be directed to achieve specified goals.The intelligence of the agents would free engineers from having to design new agents for each environment.These agents would be flexible and robust because they would be able to adapt to unanticipated aspects of the environment.
Designing such agents is a difficult problem because the environment can be almost infinitely complex.This complexity means that an agent with limited resources cannot represent and reason about the environment without describing it in a simpler form.And possibly more importantly, the complexity of the environment means that it is a challenge to generalize J. Mugan is with the School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 USA (e-mail: jmugan@cs.cmu.edu).
B. Kuipers is with the Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109 USA (e-mail: kuipers@umich.edu).from experience since each experience will be in some respect different.A solution to the difficulty of learning in a complex environment is for the agent to autonomously learn useful and appropriate abstractions.
There are approaches that do impressive learning in specific scenarios [1]- [3].But most autonomous learning methods require a discrete representation and a given set of discrete actions [4]- [8].Our goal is to enable autonomous learning to be done in a continuous environment and to enable an agent to learn its first actions.
The Problem: The world is continuous and infinitely complex (or effectively so).The dynamics of the world are also continuous, and add further complexity.An agent acts in that world by sending continuous low-level motor signals.Our goal is to show how such an agent can learn a hierarchy of actions for acting effectively in such a world.We simplify the problem of perception for our agent.Instead of dealing with continuous high-bandwidth pixel-level perception, we assume that the agent has trackers for a small number of moving objects (including its own body parts) within an otherwise static environment.These trackers provide the agent wih a perceptual stream consisting a relatively small number of continuous variables.Learning such trackers and the models that provide dynamically updated model parameters is done by methods outside the scope of this project, such as the Object Semantic Hierarchy [9].Our solution to the problem of learning a hierarchy of actions in continuous environments is the Qualitative Learner of Action and Perception, QLAP.
A. The Qualitative Learner of Action and Perception, QLAP QLAP has two major processes: (1) Modeling Events: An event is a qualitative change in the value of some state variable (Section II).QLAP starts by identifying contingencies between events: situations where the observation of one event E 1 means that another event E 2 is more likely to occur soon after.QLAP searches for improved descriptions of the conditions for contingencies, trying to find sufficiently deterministic simple models that predict the results of actions (Section III).This is done by introducing new distinctions into the qualitative abstractions of the domains of particular variables, and by identifying additional dependencies on context variables, making predictions more reliable.These improved models of the forward dynamics of the environment are represented as dynamic Bayesian networks (DBNs).This process is depicted in Figure 1.
(2) Modeling Actions: Define an action to be the occurrence of a particular event: a particular qualitative change to some variable.A reliable DBN model that leads to that event can be transformed (by familiar RL methods) into a plan for a accomplishing that consequent event by means of achieving antecedent events, and thus for carrying out that action.Since the plan thus consists of embedded actions, we get a natural hierarchical structure on actions, via the plans available to carry them out (Section IV).
A hierarchy of actions and plans must ground out in motor actions: qualitative changes to the values of motor variables that the agent can carry out simply by willing them.QLAP ensures that its higher-level actions and plans have this grounding in motor actions because it learns everything autonomously, starting from random exploration of the consequences of setting its motor variables (i.e."motor babbling").The process of modeling actions is depicted in Figure 2.
To perform exploration and learning, the two processes of Modeling Events and Modeling Actions run continuously as the agent acts in the world (Section V).Those actions can be driven initially by random motor babbling.However, they can be driven by autonomous exploration through various intrinsic motivation drives.They can also be driven by explicit coaching, or by "play" -posing various goals and making plans to achieve them.(See [10] for a video of QLAP.

B. Contributions
To the best of our knowledge, QLAP is the only algorithm that learns states and hierarchical actions through autonomous exploration in continuous, dynamic environments with continuous motor commands.For the field of Autonomous Mental Development, QLAP provides a method for a developing agent to learn its first temporally-extended actions and to learn more complex actions on top of previously-learned actions.
The related field of reinforcement learning is about enabling an agent to learn from experience to maximize a reward signal [11].QLAP addresses three challenges in reinforcement learning: (1) continuous states and actions, (2) automatic hierarchy construction, and (3) automatic generation of reinforcement learning problems.Continuous states and actions are a challenge because it is hard to know how to generalize from experience since no two states are exactly alike.There exist many function approximation methods, and other methods that use real values [3], [12]- [16], but QLAP provides a method for discretizing the state and action space so that the discretization corresponds to the "natural joints" in the environment.Finding these natural joints allows QLAP to use simple learning algorithms while still representing the complexity of the environment.
Learning of hierarchies can enable an agent to explore the space more effectively because it can aggregate smaller actions into larger ones.QLAP creates a hierarchical set of actions from continuous motor variables.Currently, most reinforcement learning problems must be designed by the human experimenter.QLAP autonomously creates reinforcement learning problems as part of its developmental progression.
Section II discusses the qualitative representation used to discretize the continuous data.Section III explains how QLAP learns models of events, and Section IV describes how models of events are converted into actions.Section V discusses how the QLAP agent explores and learns about the world.Experimental results are presented in Section VI, and the paper concludes with a discussion (Section VII), an overview of related work (Section VIII), and a summary (Section IX).

II. QUALITATIVE REPRESENTATION
A qualitative representation allows an agent to bridge the gap between continuous and discrete values.It does this by encoding the values of continuous variables relative to known landmarks [17].The value of a continuous variable can be described qualitatively.Its qualitative magnitude is described as equal to a landmark value or as in the open interval between two adjacent landmark values.Its qualitative direction of change can be increasing, steady, or decreasing.Because a landmark is intended to represent an important value of a variable where the system behavior may change qualitatively, a qualitative representation allows the agent to generalize and to focus on important events.

A. Landmarks
A landmark is a symbolic name for an point on a number line.Using landmarks, QLAP can convert a continuous variable ṽ with an infinite number of values into a qualitative variable v with a finite set of qualitative values Q(v) called a quantity space [17].A quantity space Q n } is a totally ordered set of landmark values, and is the set of mutually disjoint open intervals that L(v) defines in the real number line.A quantity space with two landmarks might be described by . This is shown in Figure 3. QLAP perceives the world through a set of continuous input variables and affects the world through a set of continuous motor variables. 2For each continuous input variable ṽ, two qualitative variables are created: a discrete variable v(t) that represents the qualitative magnitude of ṽ(t), and a discrete variable v(t) that represents the qualitative direction of change of ṽ(t).Also, a qualitative variable u(t) is created for each continuous motor variable ũ. 3 The result of these transformations is three types of qualitative variables that the agent can use to affect and reason about the world: motor variables, magnitude variables, and direction of change variables.The properties of these variables are shown in Table I.Each direction of change variable v has a single intrinsic landmark at 0, so its quantity space is Q( v) = {(−∞, 0), 0, (0, +∞)}, which can be abbreviated as Motor variables are also given an initial landmark at 0. Magnitude variables initially have no landmarks, treating zero as just another point on the number line.Initially, when the agent knows of no meaningful qualitative distinctions among values for ṽ(t), we describe the quantity space with the empty list of landmarks, {}, as Q( v) = {(−∞, +∞)}.However, the agent can learn new landmarks for magnitude and motor variables.Each additional landmark allows the agent to perceive or affect the world at a finer granularity.

B. Events
If a is a qualitative value of a qualitative variable A, meaning a ∈ Q(A), then the event A t → a is defined by A(t − 1) = a and A(t) = a.That is, an event takes place when a discrete variable A changes to value a at time t, from some other value.We will often drop the t and describe this simply as A→a.We will also refer to an event as E when the variable and value involved are not important, and we use the notation E(t) to indicate that event E occurs at time t.
For magnitude variables, A t → a is really two possible events, depending on the direction that the value is coming from.If at time t − 1, A(t) < a, then we describe this event as ↑ A t → a.Likewise, if at time t − 1, A(t) > a, then we describe this event as ↓A t →a.However, for ease of notation, we generally refer to the event as A t →a.We also say that event

III. MODELING EVENTS
There are many methods for learning predictive models in continuous environments.Such models have been learned, for example, using regression [3], [12]- [14], neural networks [15], and Gaussian processes [16].But as described in the introduction, we want to break up the environment and represent it using a qualitative representation.
In a discretized environment, dynamic Bayesian networks (DBNs) are a convenient way to encode predictive models.Most work on learning DBNs learn a network to predict each variable at the next timestep for each action, e.g.[19]- [21].However, QLAP learns the set of actions, and QLAP works in environments where events may take more than one timestep.
QLAP learns two different types of DBN models.The first type of DBN models are those that predict events on change variables (change DBNs).The second type of DBN models are those for reaching magnitude values (magnitude DBNs); Section III-E.To learn change DBNs, QLAP uses a novel DBN learning algorithm.Given the current discretization, QLAP tracks statistics on all pairs of events to search for contingencies (Section III-A) where an antecedent event leads to a consequent event.When such a contingency is found, QLAP converts it to a DBN with the antecedent event as the parent variable and the consequent event as the child variable.QLAP adds context variables to the DBN one at a time as they make the DBN more reliable.For each DBN, QLAP also searches for a new discretization that will make the DBN more reliable.This new discretization then creates new possible events and allows new DBNs to be learned.This method is outlined in Figure 4.

A. Searching for Contingencies
The search for change DBNs begins with a search for contingencies.A contingency represents the knowledge that if the antecedent event occurs, then the consequent event will soon occur.An example would be if you flip a light switch, then the light will go off.QLAP searches for contingencies by tracking statistics on pairs of events E 1 , E 2 and extracting those pairs into a contingency where the occurrence of event E 1 indicates that event E 2 is more likely to soon occur than it would otherwise.Fig. 4: (a) Do a pairwise search for contingencies that use one event to predict another.The antecedent events are along the y-axis, and the consequent events are along the x-axis, The color indicates the probability that the consequent event will soon follow the antecedent event (lighter corresponds to higher probability).When the probability of the consequent event is sufficiently high, it is converted into a contingency (yellow).(b) When a contingency is found, it is used to create a DBN.(c) Once a DBN is created, context variables are added to make it more reliable.(d) The DBN creates a self-supervised learning problem to predict when the consequent event will follow the antecedent event.This allows new landmarks to be found.Those landmarks create new events for the pairwise search.(Best viewed in color.) (The length of the time window k is learned by noting how long it takes for motor commands to be observed as changes in the world, see [18].)With these variables, we define a contingency as which represents the proposition that if the antecedent event X t → x occurs, then the consequent event Y → y will occur within k timesteps.
2) The Pairwise Search: QLAP looks for contingencies using a pairwise search by tracking statistics on pairs of events X →x and Y →y to determine if the pair is a contingency.QLAP learns a contingency E 1 ⇒ E 2 if when the event E 1 occurs, then the event E 2 is more likely to soon occur than it would have been otherwise where P r(soon(t, E 2 )) is the probability of event E 2 occurring within a random window of k timesteps.Specifically, the contingency is learned when QLAP performs this search considering all pairs of events, excluding those where 1) The consequent event is a magnitude variable (since these are handled by the models on magnitude variables as discussed in Section III-E).
2) The consequent event is on a direction of change variable to the landmark value [0] (since we want to predict changes that result in moving towards or away from landmarks).
3) The magnitude variable corresponding to the direction of change variable on the consequent event matches the magnitude variable on the antecedent event (since we want to learn how the values of variables are affected by other variables).

B. Converting Contingencies to DBNs
In this section we describe how QLAP converts a contingency of the form event(t, X→x) ⇒ soon(t, Y →y) into a dynamic Bayesian network.A dynamic Bayesian network (DBN) is a compact way to describe a probability distribution over time-series data.Dynamic Bayesian networks allow QLAP to identify situations when the contingency will be reliable.
1) Adding a Context: The consequent event may only follow the antecedent event in certain contexts, so we also want to learn a set of qualitative context variables C that predict when event Y →y will soon follow X→x.This can be represented as a DBN r of the form r = C : event(t, X→x) ⇒ soon(t, Y →y) (6) which we abbreviate to In this notation, event E 1 = X →x is the antecedent event, and event E 2 = Y →y is the consequent event.We can further abbreviate this QLAP DBN r as Figure 5 shows the correspondence between this notation and standard DBN notation.Because we consider only cases where event E 1 occurs, we can treat the conditional probability table (CPT) of DBN r as defined over the probability that event Y →y will soon follow event X→x for each qualitative value in context C. If the antecedent event does not occur, then the CPT does not define the probability for the consequent event occurring.If the antecedent event occurs, and the consequent event does follow soon after, we say that the DBN succeeds.Likewise, if the antecedent event occurs, and the consequent event does not follow soon after, we say that the DBN fails.These models are referred to as dynamic Bayesian networks and not simply Bayesian networks because we are using them to model a dynamic system.An example of a DBN learned by QLAP is shown in Figure 6.
The set C = {v 1 , . . ., v n } consists of the variables in the conditional probability table (CPT) of the DBN r = C : E 1 ⇒ E 2 .The CPT is defined over the product space [ ] 0.50 0.98 0.97 0.04 0.50

CPT
Fig. 6: An example DBN.This DBN says that if the motor value of u x becomes greater than 300, and the location of the hand, h x , is in the range −2.5 ≤ h x < 2.5, then the variable ḣx will most likely soon become [+] (the hand will move to the right).(The limits of movement of h x are −2.5 and +2.5, and so the prior of 0.5 dominates outside of that range.) Since C is a subset of the variables available to the agent, where m ≥ n is the number of variables available to the agent. 4) Notation of DBNs: We define the reliability for q ∈ Q(C) for DBN r as which is the probability of success for the DBN for the value q ∈ Q(C).These probabilities come from the CPT and are calculated using observed counts.
The best reliability of a DBN gives the highest probability of success in any context state.We define the best reliability brel(r) of a DBN r as brel(r) = max (We require 5 actual successes for q ∈ Q(C) before it can be considered for best reliability.)By increasing the best reliability brel(r) we increase the reliability of DBN r.And we say that a DBN r is sufficiently reliable if at any time brel(r) > θ SR = 0.75.The entropy of a DBN r = C : E 1 ⇒ E 2 is a measure of how well the context C predicts that event E 2 will soon follow event E 1 .Since we only consider the timesteps where event E 1 occurs, we define the entropy H(r) of a DBN r as By decreasing the entropy H(r) of DBN r, we increase the determinism of DBN r.

C. Adding Context Variables
QLAP iteratively adds context variables to DBNs to make them more reliable and deterministic.This hillclimbing process of adding one context variable at a time is inspired by the marginal attribution process in Drescher's [5] schema mechanism.By only considering the next variable to improve the DBN, marginal attribution decreases the search space.Drescher spun off an entirely new model each time a context variable was added, and this resulted in a proliferation of models.To eliminate this proliferation of models, we instead modify the model by changing the context.
QLAP initially hillclimbs on best reliability brel(r) because the DBN models will eventually be used for planning.In our experiments, we found this to be necessary to make good plans because we want to find some context state in which the model is reliable.This allows the agent to set a subgoal getting to that reliable context state.However, we also want the model to be deterministic, so after a model is sufficiently reliable, QLAP hillclimbs on reduction of entropy H(r).

D. Learning New Landmarks
Learning new landmarks allows the agent to perceive and represent the world at a higher resolution.This increase in resolution allows existing models to be made more reliable and allows new models to be learned.QLAP has two mechanisms for learning landmarks.The first is to learn a new landmark to make an existing DBN more reliable.The second is to learn a new landmark that predicts the occurrence of an event.
1) New Landmarks on Existing DBNs: QLAP learns new landmarks based on previously-learned models (DBNs).For any particular DBN, predicting when the consequent event will follow the antecedent event is a supervised learning problem.This is because once the antecedent event occurs, the environment will determine if the consequent event will occur.QLAP takes advantage of this supervisory signal to learn new landmarks that improve the predictive ability of DBNs.
For each DBN r = C : E 1 ⇒ E 2 , QLAP searches for a landmark on each magnitude and motor variable v in each open interval q ∈ Q(v).Finding this landmark is done using the information theoretic method of Fayyad and Irani [22].The best candidate is then adopted if it has sufficient information gain, and the product of the information gain and the probability of being in that interval P r(v = q) is sufficient, and the adopted landmark would improve DBN r.
2) New Landmarks to Predict Events: QLAP also learns new landmarks to predict events.QLAP needs this second landmark learning process because some events may not be preceded by another known event.An example of this is when an object moves because it is hit by another object.In this case, it needs to learn that a distance of 0 between objects is significant, because it causes one of the objects to move.
QLAP learns landmarks to predict events by storing histograms of observed real values of variables.If a landmark v * is found that co-occurs with some event E, then the agent can predict the occurrence of event E by learning a DBN of the form C : v→v * ⇒ E .Using these histograms, QLAP searches for such a landmark preceding event E by looking for a variable ṽ such that the distribution of ṽ is significantly different just before the event E than otherwise.

E. Magnitude DBN Models
A magnitude value can be less than, greater than, or equal to a qualitative value.We want to have models for a variable ṽ reaching a qualitative value q.Intuitively, if we want v = q and currently v(t) < q, then we need to set v = [+].This section describes how this process is modeled.
For each magnitude variable v and each qualitative value q ∈ Q(v), QLAP creates two models, one that corresponds to approaching the value v = q from below on the number line, and another that corresponds to approaching v = q from above.For each magnitude variable Y and each value y ∈ Q(Y ), these models can be written as DBN r + means that if Y t < y and Ẏ = [+], then eventually event Y →y will occur (DBN r − is analogous in this discussion).As the notation suggests, we can treat Ẏ →[+] ⇒ Y →y similarly to how we treat a contingency, and we can learn context variables for when this model will be reliable.These models are based on the test-operate-test-exit (TOTE) models of Miller et al. [23].Magnitude DBNs do not use the "soon" predicate because how long it takes to reach a qualitative value is determined by how far away the variable is from that value.Instead, statistics are gathered on magnitude DBNs when the agent sets Ẏ = [+] to bring about Y →y.DBN r + is successful if Y →y occurs while Ẏ = [+], and it fails if the agent is unable to maintain Ẏ = [+] long enough to bring about event Y →y.Like change DBNs, magnitude DBNs will be used in planning as described in Section IV. 5   IV.MODELING ACTIONS QLAP uses the learned DBN models to create plans for performing actions.There are two broad planning frameworks within AI: STRIPS-based goal regression [24], and Markov Decision Process (MDP) planning [25].Goal regression has 5 While context variables are learned on magnitude DBNs, experiments showed that landmarks learned from these models were not useful to the agent, so these models are not used for learning landmarks in QLAP.
the advantage of working well when only some of the variables are relevant, and MDP planning has the advantage of providing a principled framework for probabilistic actions [26].Planning in QLAP was designed to exploit the best of both frameworks.A broad principle of QLAP is that the agent should learn a factored model of the environment to make learning and planning more tractable.QLAP uses MDP planning to plan within each learned factor and uses goal regression to stitch the factors together.
QLAP defines an action to achieve each qualitative value of each variable.QLAP creates plans from the learned models, where each plan is a different way to perform an action.An action can have zero, one, or many plans.If an action has no plans it cannot be performed.If an action has multiple plans, then the action can be performed in more than one way.The actions that can be called by each plan are QLAP actions to bring about qualitative events.This stitches the plans together and leads to an action hierarchy because the plans to perform one QLAP action call other QLAP actions as if they were primitive actions.This hierarchical action network encodes all of the learned skills of the agent.See Figure 7.
Each plan is policy of an MDP created from a DBN.The policy for each plan is learned using a combination of modelbased and model-free methods.QLAP uses MDP planning instead of only goal regression because the transitions are probabilistic.Since the variables in the state space of the plan only come from the DBN model, we minimize the problem of state explosion common to MDP planning.Additionally, we use MDP planning instead of a more specialized planning algorithm such as RRT [27] because RRT and similar algorithms are designed specifically for moving in space, while the actions taken by the QLAP plan may be arbitrary, such as "hit the block off the table." In this section, we define actions and plans in QLAP.We discuss how change and magnitude DBNs are converted into plans.We then discuss how QLAP can learn when variables need to be added to the state space of a plan, and we conclude with a description of how actions are performed in QLAP.

A. Actions and Plans in QLAP
Actions are how the QLAP agent brings about changes in the world.An action a(v, q) is created for each combination of qualitative variable v and qualitative value q ∈ Q(v).An action a(v, q) is called by the agent and is said to be successful if v = q when it terminates.Action a(v, q) fails if it terminates with v = q.Statistics are tracked on the reliability of actions.The reliability of an action a is denoted by rel(a), which gives the probability of succeeding if it is called.
When an action is called, the action chooses a plan to carry it out.Each plan implements only one action, and an action can have multiple different plans where each plan is a different way to perform the action.This gives QLAP the advantage of being able to use different plans in different situations instead of having one big plan that must cover all situations.As with actions, we say that a plan associated with action a(v, q) is successful if it terminates with v = q and fails if it terminates with Models learned by QLAP can result in plans.Each plan is represented as a policy π i over an MDP M i = S i , A i , T i , R i .A Markov Decision Process (MDP) is a framework for temporal decision making [25].Since QLAP learns multiple models of the environment, QLAP learns multiple MDPs.And as with models, each MDP represents a small part of the environment.The actions used within each MDP are QLAP actions.And since these actions, in turn, use plans that call other QLAP actions, the actions and plans of QLAP are tied together, and planning takes the flavor of goal regression.
We can think of this policy π i as being part of an option o i = I i , π i , β i where I i is a set of initiation states, π i is the policy, and β i is a set of termination states or a termination function [28] .An option is like a subroutine that can be called to perform a task.Options in QLAP follow this pattern except that π i is a policy over QLAP actions instead of being over primitive actions or other options.
We use the terminology of a plan being an option because options are common in the literature, and because QLAP takes advantage of the non-Markov termination function β i that can terminate after a fixed number of timesteps.However, plans in QLAP differ from options philosophically because options are usually used with the assumption that there is some underlying large MDP.QLAP assumes no large, underlying MDP, but rather creates many little, independent MDPs that are connected by actions.Each small MDP M i created by QLAP has one policy π i .

B. Converting Change DBNs to Plans
When a DBN of the form r i = C : X → x ⇒ Y → y becomes sufficiently reliable it is converted into a plan to bring about Y → y. 6 This plan can then be called by the action a(Y, y).
This plan is based on an MDP M i .In this section, we will first describe how QLAP creates MDP M i from a DBN r i .We will then describe how QLAP learns a policy for this MDP.And finally, we will describe how this policy is mapped to an option.
1) Creating the MDP from the DBN: QLAP converts DBNs of the form r i = C : X →x ⇒ Y →y to an MDP of the form M i = S i , A i , T i , R i .The state space S i consists of the Cartesian product of the values of the variables in DBN r i .The actions in A i are the QLAP actions to bring the agent to the different states of S i .The transition function T i comes from the CPT of r i and the reliability rel(a) of different actions a ∈ A i .The reward function simply penalizes each action with a cost of 2 and gives a reward of 10 for reaching the goal of Y = y.
2) Learning a Policy for the MDP: QLAP uses three different methods to learn the policy π i for each MDP M i .(1) QLAP uses the transition function T i and the reward function R i to learn the Q-table Q i using dynamic programming with value iteration [29].The Q-table value Q i (s, a) represents the cumulative, expected discounted reward of taking action a ∈ A i in state s ∈ S i .The policy π i then follows directly from Q i because for each state s, the agent can choose action a that maximizes Q i (s, a) (or it can choose some other action to explore the world).(2) As the agent further experiences the world, policy π i is updated using the temporal difference learning method Sarsa(λ) [29].(3) And as the agent gathers more statistics, its transition model may be improved.QLAP also updates the model by occasionally running one loop of the dynamic programming.
3) Mapping the Policy to an Option: An option has the form o i = I i , π i , β i .We have described how the policy π i is learned.When an option o i is created for a DBN r i = C : X→x ⇒ Y →y , the set of initiation states I i is the set of all states.
The termination function β i terminates option o i when it succeeds (the consequent event occurs) or when it exceeds resource constraints (300 timesteps, or 5 action calls) or when the agent gets stuck.The agent is considered stuck if none of the self variables (see Section V-C for a discussion of how the agent learns which variables are part of "self") or variables in S i change in 10 timesteps.

C. Converting Magnitude DBNs into Plans
As discussed in Section III, each qualitative value y ∈ Q(Y ) on each magnitude variable Y has two DBNs that correspond to achieving the event Y →y from below and above the value Y = y, respectively.Both of these DBN models are converted into plans to achieve Y →y.And like with change DBNs, each magnitude DBN r i is converted into a plan based on an MDP M i .For MDP M i , the state space S i , the set of available actions A i , the transition function T i , and the reward function R i are computed similarly as they are for change plans.The result of this is that each action a(v, q) on a magnitude variable has two plans.There is one plan to perform the action when v < q, and another plan to perform the action when v > q.See [18] for further details.

D. Improving the State Space of Plans
The state space of a plan consists of the Cartesian product of the quantity spaces Q(v) of the variables in the model from which it was created.But what if there are variables that were not part of the model, but that are nonetheless necessary to successfully carry out the plan?To learn when new variables should be added to plans, QLAP keeps statistics on the reliability of each plan and uses those statistics to determine when a variable should be added.
1) Tracking Statistics on Plans: QLAP tracks statistics on plans the same way it does when learning models.For change DBN models, QLAP tracks statistics on the reliability of the contingency.For magnitude models, QLAP tracks statistics on the ability of a variable to reach a qualitative value if moving in that direction.For plans, QLAP tracks statistics on the agent's ability to successfully complete the plan when called.
To track these statistics on the probability of a plan o being successful, QLAP creates a second-order model The child variable of second-order DBN r o initially has an empty context, and context variables are added in as they are for magnitude and change models.The notation for these models is the same as for magnitude and change models: QLAP computes rel(o), rel(o, s) and brel(o).Therefore a plan can also be sufficiently reliable if at any time brel(o) > θ SR = 0.75.
2) Adding New Variables to the State Space: Second-order models allow the agent to identify other variables necessary for the success of an option o because those variables will be added to its context.Each variable that is added to r 2 o is also added to the state space S i of its associated MDP M i .For example, for a plan created from model r i = C : X →x ⇒ Y →y , the state space S i is updated so that (variables in more than one of C o , C, {X}, or {Y } are only represented once in S i ).For both magnitude and change options, an action a(v, q) where v ∈ Q(C o ) is treated the same way as v ∈ Q(C).

E. Performing Actions
QLAP actions are performed using plans, and these plans call other QLAP actions.This leads to a hierarchy of plans and actions.
1) Calling and Processing Actions: When an action is called, it chooses a plan and then starts executing the policy of that chosen plan.Executing that policy results in more QLAP actions being called, and this process continues until a motor action is reached.When an action a(u, q) is called on a motor variable u, then QLAP sends a random motor value within the range covered by the qualitative value u = q to the body.
This hierarchical structure of actions and plans means that multiple actions will be performed simultaneously.Each plan only keeps track of what action it is currently performing.And when that action terminates, the next action is called based on according to the policy.So as the initial action called by the agent is being processed, the path between that initial action and a motor actions continually changes.
2) Terminating Actions: An action a(v, q) terminates if v = q, in which case it succeeds.It also terminates if it fails.An action fails if (a) it has no plans, or (b) for every plan for this action, the action to bring about the antecedent event of the plan is already in the call list, or (c) its chosen plan fails.Similar to an action, a plan to bring about v = q terminates if v = q, in which case it succeeds.It also terminates if it fails.A plan to bring about v = q fails if (a) the termination function β is triggered by resource constraints, or (b) there is no applicable action in the current state, or (c) the action chosen by the policy is already in the call list, or (d) the action chosen by the policy immediately fails when it is called.

V. EXPLORATION AND LEARNING
The QLAP agent explores and learns autonomously without being given a task.This autonomous exploration and learning raises many issues.For example, how can the agent decide what is worth exploring?As the agent explores, it learns new representations.How can it keep from learning unnecessary representations and getting bogged down?Should the agent use the same criteria for learning all representations?Or should it treat some representations as especially important?And finally, can the agent learn that some parts of the environment can be controlled with high reliability and low latency so that they can be considered part of "self"?
Previous sections have explained how QLAP learns representations that take the form of landmarks, DBNs, plans, and actions.This section explains how learning in QLAP unfolds over time.We first discuss how the agent explores the environment.We then discuss developmental restrictions that determine what representations the agent learns and the order in which it learns them.We then discuss how QLAP pays special attention to goals that are hard to achieve.And finally, we discuss how the agent learns what is part of "self."

A. Exploration
The QLAP agent explores the environment autonomously without being given a task.Instead of trying to learn to do a particular task, the agent tries to learn to predict and control all of the variables in its environment.However, this raises difficulties because there might be many variables in the environment, and some may be difficult or impossible to predict or control.This section explains how the agent determines what should be explored and the best way to go about that exploration.
Initially, the agent motor babbles for 20,000 timesteps by repeatedly choosing random motor values and maintaining those values for a random number of time steps.After that point, QLAP begins to practice its learned actions.An outline of the execution of QLAP is shown in Algorithm 1.
The agent continually makes three types of choices during its exploration.These choices vary in time scale from coarse to fine.The agent chooses: a learned action a(v, q) to practice, the best plan o i for performing the action a(v, q), and the action based on policy π i for plan o i .
1) Choosing a Learned Action to Practice: One method for choosing where to explore is to measure prediction error and than to motivate the agent to explore parts of the space for which it currently does not have a good model.This form of intrinsic motivation is used in [30], [31].However, focusing attention on states where the model has poor prediction ability can cause the agent to explore spaces where learning is too difficult.
Schmidhuber [32] proposed a method whereby an agent learns to predict the decrease in the error of the model that results from taking each action.The agent can then choose the action that will cause the biggest decrease in prediction error.Oudeyer, Kaplan, and Hafner [33] apply this approach with a developing agent and have the agent explore regions of the sensory-motor space that are expected to produce the largest decrease in predictive error.Their method is called Intelligent Adaptive Curiosity (IAC).
QLAP uses IAC to determine which action to practice.After the motor babbling period of 20,000 timesteps, QLAP chooses a motor babbling action with probability 0.1, otherwise it uses IAC to choose a learned action to practice.Choosing a learned action to practice consists of two steps: (1) determine the set of applicable actions that could be practiced in the current state s, and (2) choose an action from that set.
The set of applicable actions to practice consists of the set of actions that are not currently accomplished, but could be performed.For a change action, this means that the action must have at least one plan.For a magnitude action a(v, q), this means that if v t < q then a( v, [+]) must have at least one plan (and similarly for v t > q).
QLAP chooses an action to practice by assigning a weight w a to each action a in the set of applicable actions.The action is then chosen randomly based on this weight w a .The weights are assigned using a version of Intelligent Adaptive Curiosity (IAC) [33] that measures the change in the agent's ability to perform the action over time and then chooses actions where that ability is increasing.
2) Choosing the Best Plan to Perform an Action: When an action is called, it chooses a plan to perform the action.QLAP seeks to choose the plan that is most likely to be successful in the current state.To compare plans, QLAP computes a weight w s o for each plan o in state s.To compute the weight w s o for plan o in state s, QLAP computes the product of the reliability of the DBN r that led to the plan rel(r, s) and the reliability of the second-order DBN rel(o, s) so that To choose the plan to perform the action, QLAP usesgreedy action plan selection ( = 0.05).With probability 1− , QLAP chooses the plan with the highest weight.And with probability it chooses a plan randomly.To prevent loops in the calling list, a plan whose DBN has its antecedent event already in the call list is not applicable and cannot be chosen.
3) Choosing an Action within a Plan: Recall from Section IV that QLAP learns a Q-table for each plan that gives a value for taking each action a in state s.Here again, QLAP uses -greedy selection.With probability 1 − , in state s, QLAP chooses action a that maximizes Q i (s, a), and with probability , QLAP chooses a random action.This action selection method balances exploration with exploitation [29].
Algorithm 1 The Qualitative Learner of Action and Perception (QLAP) pass motor command to robot 22: end for

B. Targeted Learning
Since QLAP creates an action for each variable and qualitative value combination, a QLAP agent is faced with many potential actions that could be learned.QLAP can choose different actions to practice based on the learning gradient, but what about the thresholds to learn predictive DBN models and plans?Some actions might be more difficult to learn than others, so it seems reasonable that the requirements for learning representations that lead to learning such actions should be loosened.
QLAP does targeted learning for difficult actions.To learn a plan for an action chosen for targeted learning, QLAP 1) Lowers the threshold needed to learn a contingency.
Recall from Section III-A2, that a contingency is learned when P r(soon(t, E 2 )|E 1 (t))−P r(soon(t, E 2 )) > θ pen = 0.05 (21) If event E 2 is chosen for targeted learning, QLAP makes it more likely that a contingency will by learned by setting θ pen = 0.02.
2) Lowers the threshold needed to learn a plan.Recall from Section IV-B that one of the requirements to convert a change DBN r into a plan is that brel(r) > θ SR = 0.75.
If event E 2 is chosen for targeted learning, QLAP makes it more likely that a DBN will be converted to a plan by setting θ SR = 0.25.This leaves the question of when to use targeted learning of actions.An event is chosen as a goal for targeted learning if the probability of being in a state where the event is satisfied is less than 0.05; we call such an event sufficiently rare.This is reminiscent of Bonarini et al. [34].They consider desirable states to be those that are rarely reached or are easily left once reached.

C. Learning What Is Part of "Self"
One step towards tool use is making objects in the environment part of "self" so that they can be used to perform useful tasks.The representation of self is straightforward in QLAP.A change variable is part of self if it can be quickly and reliably manipulated.QLAP learns what is part of self by looking for variables that it can reliably control with low latency.
Marjanovic [35] enabled a robot to identify what was part of self by having the robot wave its arm and having the robot assume that the only thing moving in the scene was itself.The work of Metta and Fitzpatrick [36] is similar but more sophisticated because it looks for optical flow that correlates with motor commands of the arm.Gold and Scassellati [37] use the time between giving a motor command and seeing movement to determine what is part of self.Our method for learning self is similar to that of Gold and Scassellati, but we learn what is part of self while learning actions.
A direction of change variable v is part of self if: 1) the average time it takes for the action to set v = [+] and v = [−] is less than k, and 2) the actions for both v = [+] and v = [−] are sufficiently reliable.where k is how long it takes for motor commands to be observed as changes in the world, see [18].

VI. EVALUATION
Evaluating autonomous learning is difficult because there is no pre-set task on which to evaluate performance.The approach we take is to first have the agent learn autonomously in an environment; we then evaluate if the agent is able to perform a set of tasks.It is important to note that during learning the agent does not know on which tasks it will be evaluated.

A. Evaluation Environment
The evaluation environment is implemented in Breve [38] and has realistic physics.Breve simulates physics using the Open Dynamics Engine (ODE) [39].The simulation consists of a robot at a table with a block and floating objects.The robot has an orthogonal arm that can move in the x, y, and z directions.The environment is shown in Figure 8, and the variables perceived by the agent for the core environment are shown in Table II.The block has a width that varies between 1 and 3 units.The block is replaced when it is out of reach and not moving, or when it hits the floor.Each timestep in the simulator corresponds to 0.05 seconds.I.e., 20 timesteps/second is equal to 1200 timesteps/minute is equal to 72, 000 timesteps per hour.See [18] for further details.
The robot can grasp the block in a way that is reminiscent of both the palmer reflex [40] and having a sticky mitten [41].The palmer reflex is a reflex that is present from birth until the age 4-6 months in human babies.The reflex causes the baby to close its hand when something touches the palm.In the sticky mittens experiments, three-month-old infants wore mittens covered with Velcro that allowed them to more easily grasp objects.
Grasping is implemented on the robot to allow it to grasp only when over the block.Specifically, the block is grasped if the hand and block are colliding, and the Euclidean 2D distance from the center of the block in the x and y directions is less than half the width of the palm, 3/2 = 1.5 units.
In addition to the core environment, QLAP is also evaluated with distractor objects.This is done using the floating extension environment, which adds two floating objects that the agent can observe but cannot interact with.The purpose of this environment is to evaluate QLAP's ability to focus on learnable relationships in the presence of unlearnable ones.The objects float around in an invisible box.The variables added to the core environment to make the floating extension environment are shown in Table III.

B. Experimental Conditions
We compare the performance of QLAP with the performance of a supervised learner on a set of tasks.The supervised learner is trained only on the evaluation tasks.This puts QLAP at a disadvantage on the evaluation tasks because QLAP is not informed of the evaluation tasks and QLAP learns more than the evaluation tasks.We hope QLAP can demonstrate developmental learning by getting better at the tasks over time, and that QLAP can do as well as the supervised learner.
Each agent is evaluated on three tasks.These are referred to as the core tasks.
1) move the block The evaluator picks a goal to move the block left ).The goal is chosen randomly based on the relative position of the hand and the block.A trial is terminated early if the agent hits the block in the wrong direction.2) hit the block to the floor The goal is to make bang = true.3) pick up the block The goal is to get the hand in just the right place so the robot can grasp the block and make T = true.A trial is terminated early if the agent hits the block out of reach.
The supervised learner is trained using linear, gradientdescent Sarsa(λ) with binary features [29] where the binary features come from tile coding.Tile coding is a way to discretize continuous input for reinforcement learning.Both the QLAP and the supervised learning agents are evaluated on the core tasks in both the core environment and the floating extension environment under three experimental conditions.1) QLAP The QLAP algorithm.
SupLrn-1 and SupLrn-10 are both used because SupLrn-1 has difficulty learning the core tasks due to high task diameter.QLAP learns autonomously for 250,000 timesteps (corresponding to about 3.5 hours of physical experience) as described in Section V.The supervised learning agents repeatedly perform trials of a particular core task for 250,000 timesteps.At the beginning of each trial, the core task that the supervised learning agent will practice is chosen randomly.The state of the agent is saved every 10,000 timesteps (about every 8 minutes of physical experience).The agent is then evaluated on how well it can do the specified task using the representations from each stored state.
At the beginning of each trial, a block is placed in a random location within reach of the agent and the hand is moved to a random location.Then, the goal is given to the agent.The agent makes and executes plans to achieve the goal.If the QLAP agent cannot make a plan to achieve the goal, it moves randomly.The trial is terminated after 300 timesteps or when the goal is achieved.The agent receives a penalty of −0.01 for each timestep it does not achieve the goal and a reward of 9.99 on the timestep it achieves the goal.(SupLrn-10 gets a penalty of −0.1 every 10th timestep it does not reach the goal and a reward of 9.99 on the timestep it reaches the goal.) Each evaluation consists of 100 trials.The rewards over the 100 trials are averaged, and the average reward is taken as a measure of ability.For each experiment, 20 QLAP agents and 20 supervised learning agents are trained.
location of second floating object in x, y, and The results are shown in Figures 9 and 10. Figure 9 compares QLAP and supervised learning on the task of moving the block in the specified direction.As can be seen in Figure 9(a), SupLrn-1 was not able to do the task well compared to QLAP due to the high task diameter (the number of timesteps needed to complete the task).Having the supervised learning agents choose an action every 10 timesteps improved their performance, as can be seen in Figure 9(b).But as can be seen by visually inspecting Figure 9(c), the performance of supervised learning degrades much more than the performance of QLAP degrades when the distractor objects are added.
This same pattern of QLAP outperforming SupLrn-1, SupLrn-10 doing as well or better than QLAP in the environment without the distractor objects, but then QLAP not   degrading with the distractor objects was also observed for the tasks of hitting the block off the table and picking up the block.Figure 10 shows the performance of QLAP and supervised learning on the tasks of hitting the block off the table and picking up the block.For brevity, this figure only contains the final comparison of QLAP and SupLrn-10 with floating objects on those tasks.We see that QLAP does better than SupLrn-10 on these more difficult tasks in the environment with floating objects.For all three tasks, QLAP displays developmental learning and gets better over time.
These results suggest the following conclusions: (1) QLAP autonomously selects an appropriate coarse temporal coding for actions, out-performing the fine-grained actions used in SupLrn-1, due to the resulting large task diameter; (2) QLAP is more robust to distractor events than SupLrn-10.

D. Additional evaluation: QLAP learns landmarks that are generally useful
We want to show that the learned landmarks really do represent the "natural joints" in the environment.Since we have no ground truth for what the natural joints of the environment are, we will compare the results of tabular Qlearning using landmarks learned with QLAP with the results of tabular Q-learning using randomly generated landmarks.If the landmarks do represent the joints in the environment, then the tabular Q-learner using learned landmarks should do better than the one using random landmarks.
1) Experimental Environment: Tabular Q-learning does not generalize well.During exploratory experiments, the state space of the core environment was so large that tabular Qlearning rarely visited the same state more than once.We therefore evaluate this claim using a smaller environment and a simple task.We use the 2D core environment where the hand only moves in two dimensions.It removes the variables of the z direction from the core environment.It subtracts u z , u U G , h z , ḣz , z BT , żBT , c x , ċx , c y , ċy , T , and bang.
2) Experimental Conditions: (a) QLAP landmarks Tabular Q-learning using landmarks learned using QLAP after a previous run of 100,000 timesteps on the 2D core environment.(b) random landmarks Tabular Q-learning using randomly generated landmarks.To generate the random landmarks, for each magnitude or motor variable v, a random number of landmarks between 0 and 5 is chosen.Each landmark is then placed in a randomly chosen location within the minimum and maximum range observed for v during a typical run of QLAP.Note that motor variables already have a landmark at 0, so each motor variable had between 1 and 6 landmarks.
3) Results: The results are shown in Figure 11.Tabular Qlearning works much better using the learned landmarks than using the random ones.

A. Description of Learned Representations
One of the first contingencies (models) that QLAP learns is that if it gives a positive force to the hand, then the hand will move to the right.But this contingency is not very reliable because in the simulator it takes a force of at least 300 units to move the hand.The agent learns a landmark at 300 on that motor variable, and modifies the model to state that if the force is at least 300, then the hand will move to the right.But this model still is not completely reliable, because if the hand is already all the way to the right, then it can't move any farther.But from this model, the agent can note the location of its hand each time it applies a force of 300, and it can learn a landmark to indicate when the hand is all the way to the right.The completed model states that if the hand is not all the way to the right and a force of at least 300 is given, then the hand will usually move to the right.Even this model is not completely reliable, because there are unusual situations where, for example, the hand is stuck on the block.But the model is probabilistic, so it can handle nondeterminism.
The agent also learns that the location of the right side of the hand in the frame of reference of the left side of the block has a special value at 0. It learns this because it notices that the block begins to move to the right when that value is achieved.It then creates a landmark to indicate that value and an action to reach that value.Based on this landmark, QLAP can learn a contingency (model) that says if the value goes to 0, then the block will move to the right.It can then learn other landmarks that indicate in which situations this will be successful.In a similar way, learns to pick up the block and knock the block off the table.

B. Theoretical Bounds
QLAP is reasonably efficient in time and space.Let V be the number of variables.QLAP searches for pairs of events to form a contingency, so this process is O(V 2 ) in both time and space.QLAP searches for context variables and landmarks for each learned contingency.It does this by considering one variable at a time for each DBN, thus these processes are O(V 3 ) in time and space.This of course means that if you had a very large number of variables, such as 1000, that the variables would need to be categorized so that only a subset of possible contingencies were considered.
MDP planning is known to be computationally expensive because the state space grows exponentially with the number of variables.However this explosion is limited in QLAP because QLAP builds many MDPs, each consisting of only a few variables.Essentially, QLAP searches for the simplest MDP models that give reasonable descriptions of the observed dynamics of the environment.The search is a greedy breadthfirst search, incrementally increasing the number of variables in the MDP (via the DBN).Note that any given MDP model describes the world in terms of a certain set of explicit variables.The rest of the dynamics of the world are encoded in the probabilities in the CPT tables.Presumably, there is a general trade-off between the number of variables in the model and the determinism of the CPT.However, QLAP is based on the assumption that that general trade-off is dominated by the choice of the right variables and the right landmarks for those variables.That is, the right small models may well be quite good, and the hill-climbing methods of QLAP can often find them.

C. Assumptions of QLAP
QLAP assumes that any goal that an outside observer would want the agent to accomplish is represented with an input variable.QLAP also assumes that meaningful landmarks can be found on single variables.In some cases when these assumptions are violated, QLAP can do a search on combinations of variables (see [18]).
QLAP also assumes a set of continuous motor primitives that correspond to orthogonal directions of movement.QLAP builds on the work of Pierce and Kuipers [42].In their work, an agent was able to use principal components analysis (PCA) [43] to learn a set of motor primitives corresponding to turn and travel for a robot that had motors to turn each of two wheels independently.

VIII. RELATED WORK
QLAP learns states and hierarchical actions in continuous, dynamic environments with continuous motors through autonomous exploration.The closest direct competitor to QLAP is the work of Barto, Jonsson, and Vigorito.Given a DBN model of the environment, the VISA algorithm [44] creates a causal graph which it uses to identify state variables for options.Like QLAP, the VISA algorithm performs state abstraction by finding the relevant variables for each option.Jonsson and Barto [20] learn DBNs through an agent's interaction with a discrete environment by maximizing the posterior of the DBN given the data by building a tree to represent the conditional probability.Vigorito and Barto [45] extends [20], [44] by proposing an algorithm for learning options when there is no specific task.This work differs from QLAP in that learning takes place in discrete environments with events that are assumed to occur over one-timestep intervals.The work also assumes that the agent begins with a set of discrete actions.Because QLAP is designed for continuous environments with dynamics, QLAP uses a qualitative representation.This qualitative representation leads to a novel DBN learning algorithm for learning predictive models, and a novel method for converting those models into a set of hierarchical actions.
Shen's LIVE algorithm [46] learns a set of rules in firstorder logic and then uses goal regression to perform actions.The algorithm assumes that the agent already has basic actions, and the experiments presented are in environments without dynamics such as the Tower of Hanoi.Another method for learning planning rules in first-order logic is [7], [8].The rules they learn are probabilistic, given a context and an action their learned rules provide a distribution over results.This algorithm assumes a discrete state space and that the agent already has basic actions such as pick up.
QLAP's structure of actions and plans is reminiscent of the MAXQ value function decomposition [47].QLAP defines its own actions and plans as it learns, and as the agent learns a more refined discretization the hierarchy changes.There has also been much work on learning hierarchy.Like QLAP, Digney [48] creates a task to achieve each discrete value of each variable.However, QLAP learns the discretization.Work has been done on learning a hierarchical decomposition of a factored Markov decision process by identifying exits.Exits are combinations of variable values and actions that cause some state variable to change its value [44].Exits roughly correspond to the DBNs found by QLAP except that there is no explicit action needed for QLAP DBNs.Hengst [49] determined an order on the input variables based on how often they changed value.Using this ordering, he identified exits to change the next variable in the order and created an option for each exit.
There has been other work on structure learning in sequential decision processes where the environment can be modeled as a factored MDP.Degris et al. [19] proposed a method called SDYNA that learns a structured representa-tion in the form of a decision tree and then uses that structure to compute a value function.Strehl et al. [21] learn a DBN to predict each component of a factored state MDP.Hester and Stone [50] learn decision trees to predict both the reward and the change in the next state.All of these methods are evaluated in discrete environments where transitions occur over onetimestep intervals.

IX. SUMMARY AND CONCLUSION
The Qualitative Learner of Action and Perception (QLAP) is an unsupervised learning algorithm that allows an agent to autonomously learn states and actions in continuous environments.Learning actions from a learned representation is significant because it moves the state of the art of autonomous learning from grid worlds to continuous environments.Another contribution of QLAP is providing a method for factoring the environment into small pieces.Instead of learning one large predictive model, QLAP learns many small models.And instead of learning one large plan to perform an action, QLAP learns many small plans that are useful in different situations.
QLAP starts with a bottom-up process that detects contingencies and builds DBN models to identify the conditions (i.e., values of context variables) under which there are neardeterministic relations among events.Meanwhile, an action is defined for each event, where the action has the intended effect of making that event occur.The desirable situation is for an action to have one or more sufficiently reliable plans for implementing the action.If there are no plans for an action, or if the plans are not sufficiently reliable, the action is still defined, but it is not useful, so it will not be used as a step in a higher-level plan.Each plan comes from a sufficiently reliable DBN, where the overall structure is that DBNs lead to MDPs, MDPs are converted into policies, and policies are plans.
QLAP explores autonomously and tries to learn to achieve each qualitative value of each variable.To explore, the agent continually chooses an action to practice.To choose which action to practice, QLAP uses Intelligent Adaptive Curiosity (IAC).IAC motivates the agent to practice actions that it is getting better at, and IAC motivates the agent to stop practicing actions that are too hard or too easy.
QLAP was evaluated in environments with simulated physics.The evaluation was performed by having QLAP explore autonomously and then measuring how well it could perform a set of tasks.The agent learned to hit a block in a specified direction and to pick up the block as well or better than a supervised learner trained only on the task.The evaluation also showed that the landmarks learned by QLAP were broadly useful.Future work will consist of incorporating continuous learning methods within the discretized representation learned by QLAP.This should enable QLAP to leverage both best of discrete learning and the best of continuous learning.

Fig. 3 :
Fig. 3: Landmarks divide the number line into a discrete set of qualitative values.

Fig. 7 :
Fig. 7: Planning in QLAP.(a) QLAP defines an action for each qualitative value of each variable.This action is to bring variable Y to value y.(b) Each action can have multiple plans, which are different ways to perform the action.Each plan comes from an MDP.The valueQ i (s, a)is the cumulative, expected, discounted reward of choosing action a in state s in MDP M i .Given the function Q i , the policy π i can be computed.(c) Plans are created from models.The state space for an MDP is the cross product of the values of X, Y , Z, and W from the model (although more can be added if needed).(d) The actions for each plan are QLAP actions to move to different locations in the state space of the MDP.This is reminiscent of goal-regression.Above, we see that one of the actions for M i is to call the QLAP action to bring about X→x.This link results from event X→x being the antecedent event of the DBN model to bring about event Y →y.

10 Fig. 9 :
Fig. 9: Moving the block.(a) QLAP does better than SupLrn-1 because of the high task diameter.(b) SupLrn-10 does better than QLAP.(c) When the floating objects are added, the performance of SupLrn-10 degrades much more than the performance of QLAP degrades.

Fig. 10 :
Fig. 10: QLAP outperforms supervised reinforcement learning using tile coding on the more difficult tasks in the floating extension environment.(a) Knock the block off the table.(b) Pick up the block.

Fig. 11 :
Fig.11: QLAP landmarks enable the agent to learn the task better than do random landmarks.

TABLE I :
Types of Qualitative Variables QLAP notation DBN representation Fig. 5: Correspondence between QLAP DBN notation and traditional graphical DBN notation.(a) QLAP notation of a DBN.Context C consists of a set of qualitative variables.Event E 1 is an antecedent event and event E 2 is a consequent event.(b) Traditional graphical notation.Boolean parent variable event(t, E 1 ) is true if event E 1 occurs at time t.Boolean child variable soon(t, E 2 ) is true if event E 2 occurs within k timesteps of t.The other parent variables are the context variables in C. The conditional probability table (CPT) gives the probability of soon(t, E 2 ) for each value of its parents.For all elements of the CPT where event(t, E 1 ) is false, the probability is undefined.The remaining probabilities are learned through experience.
2 o is succeeds(t, o), which is true if option o succeeds after being called at time t and is false otherwise.The parent variables of r 2 o are call(t, o) and the context variables in C o .The Boolean variable call(t, o) is true when the option is called at time t and is false otherwise.When created, model r 2

TABLE II :
Variables of the core environment ṪL location of nearest edge of block in x direction in coordinate frame defined by left edge of table T R , ṪR location of nearest edge of block in x direction in coordinate frame defined by right edge of table T T , ṪT location of nearest edge of block in y direction in coordinate frame defined by top edge of table cx, ċx location of hand in x direction relative to center of block cy, ċy location of hand in y direction relative to center of block T block is grasped, true or false.Becomes true when the hand is touching the block and the 2D distance between the center of the hand and the center of the block is less than 1.5.bang true when block hits the floor

TABLE III :
Variables added to the core environment to make up the floating extension environment