From Structured Task Instructions to Robot Task Plans

Jianmin Ji and Xiaoping Chen

Multi-Agent Systems Lab, School of Computer Science and Technology

University of Science and Technology of China, 230026, Hefei, China

Keywords:

Causal Theory, Open Knowledge, Task Planning, Task Instruction, Robot.

Abstract:

For the purpose of allowing an autonomous robot to use task instructions for task planning, we present a for-

malization for specifying structured task instructions and provide an approach for integrating these instructions

with robot’s built-in knowledge to compute plans for open-ended tasks. We have implemented a prototype of

the system. We also report a case study of the effectiveness of the approach.

1 INTRODUCTION

To perform everyday manipulation tasks in house-

holds, a service robot needs to automatically gener-

ate sequences of primary actions adapted to the en-

vironment and tasks, which is commonly done us-

ing AI task planning methods (McDermott, 1992),

like Causal Theories (McCain and Turner, 1997),

FLUX (Thielscher, 2005), or various planning sys-

tems based on PDDL (Ghallab et al., 1998). Origi-

nally, these approaches heavily rely on having a com-

plete and certain description of the situation that the

robot is faced with, which normally oversimpliﬁes

its real counterpart when the robot performs in the

daily world (McDermott, 1992): Actions might have

stochastic effects, the current state of the world might

not be fully known, and action sequences are not ex-

pressive enough to specify competent robot manipu-

lation behaviors. To overcome these challenges, Con-

tinual Planning (Brenner and Nebel, 2009) is pro-

posed such that robots perform a continuous loop be-

tween planning, plan execution, and execution mon-

itoring, where “classical” planning can be used for

the most likely situations while the discrepancies be-

tween “physical reality” and “mental reality” can later

be recovered during the execution monitoring phase.

AI planning methods can also be integrated with com-

puter simulations to enable robots to realistically pre-

dict real-world behaviors (Johnston and Williams,

2008).

On the other hand, everyday manipulation tasks

are often open-ended, which requires the robot to in-

fer by itself how to accomplish these tasks properly.

One way to meet this requirement is to provide robots

with the ability to understand and follow the task in-

structions that are deﬁned by human users (Burgard

et al., 1999; Rybski et al., 2007; Tenorth et al., 2010;

Cantrell et al., 2012). A robot can ﬁnd and extract

knowledge of these instructions from human-robot di-

alogue, as well as web sources such as open knowl-

edge bases like Cyc

and Open Mind Indoor Com-

mon Sense (OMICS)

. Then, after necessary conver-

sions, the structured task instructions are represented

in the robot and made use of to accomplish open-

ended tasks.

Continuing both lines of research, we consider

how to compute a robot executable plan for a task in

an AI planning system following structured task in-

structions. In particular, we present a formalization

for specifying structured instructions in a robot and

provide an approach for integrating instructions with

robot’s built-in knowledge to compute plans for tasks.

In the authors’ observation, for task planning,

user-deﬁned task instructions separate themselves

from high-level task speciﬁcations written by de-

signers in two major aspects. Firstly, two differ-

ent types of knowledge, named functional and pro-

cedural knowledge, for task accomplishment are ar-

bitrarily introduced in user-deﬁned instructions and

the robot may need to integrate both types of knowl-

edge to compute plans for some tasks. Speciﬁcally,

functional knowledge speciﬁes the expected effects

of a task. Given functional knowledge, the robot

could consider these expected effects as goals and use

AI planning approaches to compute a plan (Galindo

et al., 2008). Procedural knowledge speciﬁes a plan

skeleton of how to accomplish a task. Given proce-

http://www.cyc.com/

http://commons.media.mit.edu/en/

237

Ji J. and Chen X..

From Structured Task Instructions to Robot Task Plans.

DOI: 10.5220/0004545402370244

In Proceedings of the International Conference on Knowledge Engineering and Ontology Development (KEOD-2013), pages 237-244

ISBN: 978-989-8565-81-5

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

dural knowledge, the robot needs to ﬁnd out a possi-

ble sequence of actions which meets the restrictions

of the plan skeleton to accomplish the task (Burgard

et al., 1999; Tenorth et al., 2010). Normally, both

knowledge are arbitrarily introduced in user-deﬁned

task instructions. For example, the task “heat food

in microwave” can be instructed to take the follow-

ing three steps: “place the food in microwave”, “turn

on microwave oven for two minutes”, “take the food

out”. Note that, this instruction denotes some pro-

cedural knowledge, while the ﬁrst and the third step

denotes some functional knowledge, whose expected

effects are “the food is placed inside the microwave”

and “the food is placed outside the microwave” sepa-

rately.

Secondly, user-deﬁned task instructions may miss

some details that are indispensable for robots’ execu-

tion. For example, in OMICS, there is an instruction

specifying how to “get food from refrigerator” in ﬁve

steps. The last two steps were speciﬁed as “pick food

up” and “close door”. However, if the robot has only

one arm, it must put the food somewhere before clos-

ing the door. This action is missing from the instruc-

tion.

We address these two aspects in this paper. We

present an internal representation of both functional

and procedural knowledge of task instructions and

propose a uniﬁed frameworktointegrate the both with

robot’s built-in knowledge to compute plans for open-

ended tasks. The planning program could also au-

tomatically detect the missing details, and add some

proper actions to make the computed plan executable.

A prototype of the system has been implemented

based on Answer Set Programming (ASP), a logic

programming language with Prolog-like syntax un-

der stable model semantics (Baral, 2003). We choose

ASP to implement the system because it can be use to

specify causal theories easily and it has many efﬁcient

solvers. We also report a case study of the effective-

ness of the approach.

A number of works in task planning have tackled

the challenge of understanding and utilizing task in-

structions. (Gupta and Hennacy, 2005) presents a sys-

tem which can ﬁll-in missing information from task

instructions using commonsense reasoning. (Kress-

Gazit et al., 2007) speciﬁes how to translate robot

behaviors from a description in structured English to

actual robot controllers via Linear Temporal Logic.

(Cantrell et al., 2012) demonstrates a robot planner

that can utilize natural language commands from un-

trained users. However, in these works, the internal

representations of task instructions could not handle

both functional and procedural knowledge, and the

planning system could not ﬁll-in missing steps be-

tween two successive operations in the instruction.

Other related work (Tellex et al., 2011; Chen and

Mooney, 2011; MacMahon et al., 2006; Tenbrink

et al., 2010) focus on beneﬁting route instructions for

robot navigation. Golog and its variants (Levesque

et al., 1997; Beetz et al., 2010; Son et al., 2001) can

represent and reason with both functional and proce-

dural knowledge. However, they cannot reason about

missing details that are indispensable for robots’ exe-

cution.

The rest of the paper is organized as follows. Sec-

tion 2 reviews a formalization of a task planning sys-

tem. Section 3 reports a formalization for repre-

senting functional and procedural knowledge of task

instructions. Section 4 provides an uniﬁed frame-

work to integrate both knowledge with robot’s built-in

knowledge to compute plans, and describes an imple-

mentation of the system. At last, a case study is re-

ported in Section 5 and conclusions are given in Sec-

tion 6.

2 A TASK PLANNING SYSTEM

BASED ON CAUSAL THEORIES

Without loss of generality, we choose the planning

approach based on McCain and Turner’s causal the-

ories (McCain and Turner, 1997) for robot task plan-

ning. The idea is to specify the action domain and the

planning problem in causal theories such that a causal

model correspondsto a planning solution, then use so-

phisticated AI tools or solvers to compute robot plans

for tasks. In the following, we brieﬂy review the def-

inition of causal theories, describe how to specify the

action domain and the planning problem in causal the-

ories, and sketch an implementation of the task plan-

ning system.

Firstly, the underlying propositional signature is

consisted with three pairwise-disjoint sets: a set of

action names, a nonempty set of ﬂuent names, and a

nonempty set of time names. The action-atoms are

expressions of the form a

and the ﬂuent-atoms are

expressions of the form f

, where a, f, and t are ac-

tion, ﬂuent, and time names, respectively. Atoms of

the language are either action-atoms or ﬂuent-atoms.

Specially, ⊥ denotes contradiction. Intuitively, a

is true iff the action a occurs at time t, and f

true iff the ﬂuent f holds at time t. For example,

grasp(bottle)

is an action-atom stands that the action

“grasp bottle” occurs at time 1 and holding(bottle)

is a ﬂuent-atom stands that the ﬂuent “holding bottle”

is true at time 2. A literal is either an atom a or the

negation ¬a. Formulas are formed from atoms using

propositional connectives, while ﬂuent-formulas are

KEOD2013-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

238

formed from ﬂuent-atoms.

A causal theory is a set of causal rules of the

form: φ ⇒ ϕ, where φ and ϕ are formulas. Intuitively,

the causal rule reads as “ϕ is caused if φ is true”. As a

syntax sugar, a causal rule with variables is viewed as

shorthand for the set of its ground instances, that is,

for the result of substituting corresponding variable-

free terms for variables in all possible ways.

An interpretation I is a set of literals such that

for each atom a in the language, either a ∈ I or

¬a ∈ I but not both. Given a causal theory T

and an interpretation I, the reduction T

= {ϕ |

for some φ ⇒ ϕ ∈ T and I |= φ}. T

is a propositional

theory. We say that I is a causal model of T if I is the

unique model of T

An action domain contains the knowledge of ac-

tions of the robot and changes of the environment,

which is an essential part of robots’ built-in knowl-

edge. A causal theory for an action domain will typi-

cally contain rules specifying the initial state and how

ﬂuents are changed as the result of performing an ac-

tion. We take the action grasp and corresponding ﬂu-

ents for instance.

• grasp(X): the action of griping the object X and

picking it up.

• holding(X): the ﬂuent that the object X is held in

the grip of the robot.

• on(X,Y): the ﬂuent that the object X is on the

object Y.

In addition, σ is a meta-variable ranging over

{on(X,Y), ¬on(X,Y), holding(X), ¬holding(X)}.

The effect of executing the action grasp(X) is de-

scribed as follows:

grasp(X)

⇒ holding(X)

t+1

(1)

grasp(X)

∧ on(X,Y)

⇒ ¬on(X,Y)

t+1

(2)

The precondition of grasping requires the grip holds

nothing:

grasp(X)

∧ holding(Y)

⇒ ⊥ (3)

The occurrence of the action is exogenous to the

causal theory:

grasp(X)

⇒ grasp(X)

(4)

¬grasp(X)

⇒ ¬grasp(X)

(5)

The initial state (at time 0) can be arbitrary:

⇒ σ

(6)

The frame problem is overcome by the following “in-

ertia” rules:

∧ σ

t+1

⇒ σ

t+1

(7)

We deﬁne a state s for time t as a set of ﬂuent-

atoms with the time name t. Intuitively, s denotes

a world speciﬁed by the ﬂuents that are true at a

time step. Given a causal theory with time names

{0, 1, . . . , n}, we can deﬁne a trajectory as a sequence

, α

, s

, . . . , α

n−1

, s

i, where s

is a state for time i

(0 ≤ i ≤ n), and α

is an action-atom (0 ≤ j < n).

Note that, a causal model of such a causal theory

contains exactly a trajectory of the above form, i.e.,

∪{α

}∪ ·· · ∪{α

n−1

}∪ s

is a causal model of such

a causal theory. We also call it a trajectory of the

causal theory.

Given a description of the goals to be completed,

we could use a ﬂuent-formula ϕ

formed by ﬂuent-

atoms with the last time name n to specify the re-

quirements of the goal states. Then the task planning

problem is reduced to the causal theory for the action

domain by adding the causal rule ¬ϕ

⇒ ⊥. Clearly,

a causal model or a trajectory of the causal theory cor-

responds to a solution of the planning problem.

A task planning system based on causal theories

has been implemented on our service robot (Chen

et al., 2010; Chen et al., 2012). A logic pro-

gramming language named Answer Set Program-

ming (ASP) (Baral, 2003) is chosen for the calcu-

lation of causal theories and an efﬁcient ASP solver

iclingo (Gebser et al., 2008) is used for computing

task plans. More details can be found in our robot’s

description paper

3 A FORMALIZATION FOR

STRUCTURED TASK

INSTRUCTIONS

Both functional and procedural knowledge are com-

monly used in everyday task instructions. For exam-

ple, an instruction written by web users can be found

in OMICS:

Heat bread in microwave involves the steps:

1. place the bread in the microwave;

2. turn on microwave oven for two minutes;

3. check temperature;

4. repeat till desired temperature is achieved.

Some steps, like “turn on microwave oven for two

minutes”, could be directly mapped to primitive ac-

tions which denote operating routines of the robot.

Others may denote sub-tasks whose instructions

could be speciﬁed by other functional or procedural

http://ai.ustc.edu.cn/en/robocup/atHome/ﬁles/

WEHome2013TDP.pdf

FromStructuredTaskInstructionstoRobotTaskPlans

239

knowledge. For instance, “place the bread in the mi-

crowave” is also a task, whose expected effect is “the

bread is placed in the microwave”.

There has been some work on translating natural

language sentences into logical representations (Baral

et al., 2011) and handling the symbol grounding prob-

lem (Vogt, 2006). Our previous work (Xie et al.,

2012; Chen et al., 2012) also provided an approach on

converting semi-structured natural language instruc-

tions to structured representations. Here we focus

on how to represent both functional and procedural

knowledge of structured task instructions.

Firstly, the underlying signature is consisted with

three pairwise-disjoint sets: a set of action names,

a set of ﬂuent names, and a set of task names. A

ﬂuent-speciﬁcation is formed from ﬂuent names us-

ing propositional connectives. A procedure is deﬁned

recursively as follows:

• an action name a is a procedure,

• a ﬂuent-speciﬁcation F is a procedure,

• a task name T is a procedure,

• if P

(1 ≤ i ≤ m) are procedures then P

;. . . ;P

a procedure,

• if P

(1 ≤ i ≤ m) are procedures then P

|· ·· |P

a procedure,

• if P

and P

are procedures and F is a ﬂuent-

speciﬁcation then if F then P

else P

is a proce-

dure,

• if P is a procedure and F is a ﬂuent-speciﬁcation

then while F do P is a procedure.

Intuitively, a ﬂuent-speciﬁcation represents a situa-

tion of the world and a procedure represents a skele-

ton of an execution program. For example, P

;. . . ;P

represents a procedure executed from P

to P

step

by step and P

|· ·· |P

represents a nondeterministic

choice from P

, . . . , P

A piece of functional knowledge is a pair (T, F)

where T is a task name and F is a ﬂuent-speciﬁcation

and a piece of procedural knowledge is a pair (T, P)

where P is a procedure. At last, we formalize a struc-

tured task instruction as a pair (F , P ), where F is a

set of functional knowledge and P is a set of proce-

dural knowledge.

Continuing the example in the beginning of the

section, a structured task instruction (F , P )

could be

extracted from the steps as

F = { (Place(bread, micro), in(bread, micro))},

P = { (Heat(bread, micro), P)},

The meaning of action, ﬂuent, task names would be

explained in Section 5.

where P is the procedure:

Place(bread, micro);turnon(micro);check(bread);

while¬isok(bread)

do ( Place(bread, micro);turnon(micro);check(bread)),

Heat(bread, micro) and Place(bread, micro) are task

names as they intimated, in(bread, micro) and

isok(bread) are ﬂuent names, and others are action

names.

Given a robot, the structured task instruction col-

lects its both functional and procedural knowledge of

how to accomplish tasks. Intuitively, a piece of func-

tional knowledge (T, F) means the task T would be

accomplished when the ﬂuent-speciﬁcation F is satis-

ﬁed at some time point, a piece of procedural knowl-

edge (T, P) means that T would be accomplished

when the robot has performed the procedure P. Now

we make it precise in the task planning system based

on causal theories.

Let Π be a causal theory specifying the action

model of the robot and τ = hs

, α

, s

, . . . , α

n−1

, s

be a trajectory of Π, we deﬁne τ satisﬁes a ﬂuent-

speciﬁcation F if for some 0 ≤ i ≤ n, s

|= F

where

is the ﬂuent-formula obtained from F by replacing

each occurred ﬂuent name f by the ﬂuent-atom f

. τ

satisﬁes a procedure P is deﬁned recursively as fol-

lows:

• If P = a, where a is an action name, then a is the

action name occurred in the action-atom α

;

• If P = F, where F is a ﬂuent-speciﬁcation, then

|= F

;

• If P = T, where T is a task name, then τ satis-

ﬁes some procedural knowledge P

′

of T or some

functional knowledge F

′

of T;

• If P = P

;. . . ;P

, where P

(1 ≤ i ≤ m) are pro-

cedures, then there exists 0 ≤ n

≤ n

≤ ··· ≤

m−1

≤ n such that:

– the trajectory hs

, α

, . . . , s

i satisﬁes P

;

– the trajectory hs

, α

, . . . , s

i satisﬁes P

;

·· ·

– the trajectory hs

m−1

, α

m−1

, . . . , s

i satisﬁes P

;

• If P = P

|· ·· |P

, where P

(1 ≤ i ≤ m) are proce-

dures, then τ satisﬁes P

for some 1 ≤ j ≤ m;

• If P = if F then P

else P

, where P

and P

are

procedures and F is a ﬂuent-speciﬁcation, then τ

satisﬁes F implies τ satisﬁes P

otherwise τ satis-

ﬁes P

• If P = while F do P

, where P

is a procedure and

F is a ﬂuent-speciﬁcation, then τ does not satisfy

F or there exists 0 ≤ n

≤ n such that:

– the trajectory hs

, α

, . . . , s

i satisﬁes P

, and

KEOD2013-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

240

– the trajectory hs

, α

, . . . , s

i satisﬁes P.

Intuitively, τ satisﬁes a if the action a is executed at

time 0; τ satisﬁes F if the situation condition F is true

at the initial state; τ satisﬁes P

;. . . ;P

if τ satisﬁes

the sequence of procedures P

, . . . , P

continuously; τ

satisﬁes P

|· ·· |P

if τ stochastically satisﬁes a proce-

dure P

(1 ≤ i ≤ m); and τ satisﬁes the conditional and

the loop procedures are explained as usual.

At last, given a piece of functional knowledge

(T, F) and a piece of procedural knowledge (T, P), let

Π be a causal theory for the action model of the robot,

if there exists a trajectory τ of Π such that τ satisﬁes

F or P, then the task T would be accomplished if the

robot performed the trace speciﬁed by τ.

4 INTEGRATING STRUCTURED

TASK INSTRUCTIONS FOR

PLANNING

In this section, we consider how to compute a robot

plan for tasks with the help of structured task instruc-

tions. Following the notions used in previous sec-

tions, an extended task planning problem ∆ is a tu-

ple (Π, T , F , P ), where Π is the causal theory for

the action domain, T is a set of task names that

need to be accomplished, and (F , P ) is a struc-

tured task instruction. A sequence of action names

, . . . , a

i is a plan of ∆, if there exists a trajectory

τ = hs

, α

, s

, . . . , α

, s

i of Π such that

• the action name a

occurs in the action-atom α

for

each 1 ≤ i ≤ m and

• for each task T ∈ T , either of following two con-

ditions is true

– there exists a pair (T, F) ∈ F such that τ satis-

ﬁes F; or

– there exists a pair (T, P) ∈ P such that τ satisﬁes

Clearly, the robot could execute a plan in its action

domain and accomplish the required tasks after the

execution.

Now we provide an approach for computing plans

of extended task planning problems by converting

structured task instructions into causal rules.

For each pair (T, F) ∈ F of an extended task plan-

ning problem ∆ = (Π, T , F , P ), we use tr(T, F) to de-

note the set of the causal rule:

⇒ T

where t is a meta-variable ranging over time names of

Π. For each pair (T, P) ∈ P of ∆, we use tr

∗

(P) to

denote the set of causal rules: ¬p

⇒ ¬p

where p is

a new task name for the procedure P and

• If P = a where a is an action name, then

⇒ p

• If P = F where F is a ﬂuent-speciﬁcation, then

⇒ p

• If P = T

′

where T

′

is a task name, then

′

⇒ p

• If P = P

;. . . P

where P

(1 ≤ i ≤ m) are proce-

dures, then tr

∗

) for each 1 ≤ i ≤ m and

∧ ··· ∧ p

⇒ p

where p

is the new task name for the procedure

for each 1 ≤ i ≤ m, t

, . . . , t

are also meta-

variables for time names such that t

≤ · ·· ≤ t

≤

• If P = P

|· ·· |P

where P

(1 ≤ i ≤ m) are proce-

dures, then for each 1 ≤ i ≤ m, tr

∗

) and

⇒ p

• If P = if F then P

else P

where P

and P

are

procedures and F is a ﬂuent-speciﬁcation, then

∗

(F;P

), tr

∗

(¬F;P

), and

⇒ p

where ρ

is the new task name for the procedure

F;P

and ρ

is the new task name for the proce-

dure ¬F;P

• If P = while F do P

, where P

is a procedure and

F is a ﬂuent-speciﬁcation, then tr

∗

(F), tr

∗

and

¬ρ

⇒ p

∧ p

⇒ p

where ρ is the new task name for the procedure F,

is the new task name for the procedure P

, and

is the meta-variable for time names such that

≤ t.

In fact, the above conversion directly explains the se-

mantics of a procedure in causal rules. Then we use

tr(T, P) to denote the set of causal rules: tr

∗

(P) and

⇒ T

At last, with a slight abuse of the symbol, we use tr(∆)

to denote the set of causal rules combined with:

tr(T, F) for each (T, F) ∈ F

tr(T, P) for each (T, P) ∈ P

¬T

∧ ··· ∧ ¬T

⇒ ⊥ for each T ∈ T

FromStructuredTaskInstructionstoRobotTaskPlans

241

where time names of Π are 0, . . . , n.

The following proposition shows that a causal

model of tr(∆) corresponds to a plan of ∆.

Proposition 1. Given a causal theory of an action do-

main with time names {0, . . . , n}, a sequence of action

names ha

, . . . , a

i is a plan of an extended task plan-

ning problem ∆ w.r.t. the action domain, if and only

if, there exists a causal model S of tr(∆) such that for

each 1 ≤ i ≤ n, the action-atom a

i−1

occurs in S.

As discussed in Introduction, there are mainly two

difﬁculties in applying structured task instructions for

planning: Functional and procedural knowledge need

to be integrated in a uniform framework for com-

puting task plans, and indispensable details might

be missing between two successive steps in a proce-

dure. In our approach, both functional and procedural

knowledge are converted to causal rules, then they are

used in the uniform reasoning mechanism of causal

theories and worked together to compute plans for re-

quired tasks. On the other hand, if an action cannot be

executed in a procedure, then based on the formaliza-

tion of the action domain, some proper actions would

be automatically added to achieve preconditions of

the action and make the computed plan executable.

We have implemented a prototype of the sys-

tem for computing plans of extended task planning

problems and run experiments on many instructions

Again, ASP is used for computing causal models of

causal theories. (Ferraris, 2007) speciﬁed a polyno-

mial cost conversion from a causal theory to an ASP

program such that a causal model corresponds to an

answer set. In the prototype, an extended task plan-

ning problem ∆ is ﬁrstly converted to the causal the-

ory tr(∆), then tr(∆) is converted to its corresponding

ASP program whose answer sets are computed by a

sophisticated ASP solver iclingo. At last, plans of ∆

are extracted from these computed answer sets.

5 A CASE STUDY

In this section, we illustrate our approach of integrat-

ing structured task instructions for planning on the

running example “Heat bread in microwave”. Given

the structured task instruction (F , P ) described in

Section 3, we could create an extended task planning

∆ = (Π, {Heat(bread, micro)}, F , P ) where Π is the

causal theory for the action domain of the robot. In

particular, Π

is a causal theory with action names:

• move(L): move to the location L.

http://ai.ustc.edu.cn/en/robocup/atHome/aspcontrol/

A complete speciﬁcation can be found at http://ai.ustc.

edu.cn/en/robocup/atHome/aspcontrol/example.html.

• grasp(bread): grip the bread and picking it up.

• putdown(bread): put bread down.

• open(micro): open the door of the microwave.

• close(micro): close the door of micro.

• turnon(micro): turn on micro for two minutes.

• putin(bread, micro): put bread in micro.

• takeout(bread, micro): take bread out of micro.

• check(bread): check the temperature of bread.

• wait: wait for two minutes.

ﬂuent names (some of them):

• location(X, L): the object X is located at L.

• in(bread, micro): bread is in micro.

• isok(bread): temperature of bread is acceptable.

task names:

• Heat(bread, micro): heat bread in micro.

• Place(bread, micro): place bread in micro.

Initially, the robot, bread and microwave are lo-

cated at different places, i.e., location(robot,1),

location(bread, 2), location(micro, 3). The door of

micro is closed and bread needs to be heated three

times in micro to reach the required temperature.

As proposed in the previous section, the extended

task planning problem ∆ can be converted to a causal

theory tr(∆):

Π ∪ tr(Place(bread, micro), in(bread, micro)) ∪

tr( Heat(bread, micro), P) ∪

{Heat(bread, micro)

∧· ··∧Heat(bread, micro)

⇒ ⊥}

In particular, tr( Place(bread, micro), in(bread, micro) )

contains the causal rule:

in(bread, micro)

⇒ Place(bread, micro)

where t is a meta-variable ranging over time names of

Π. tr(Heat(bread, micro), P) contains causal rules

¬p

∗

⇒ ¬p

∗

¬p

⇒ ¬p

¬p

⇒ ¬p

Place(bread, micro)

∧ turnon(micro)

∧ check(bread)

⇒ p

∗

isok(bread)

⇒ p

∗

∧ p

⇒ p

Place(bread, micro)

∧ turnon(micro)

∧ check(bread)

∧ p

⇒ p

Due to the limit of space, these causal rules are slightly

simpliﬁed.

KEOD2013-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

242

where p

∗

, p

, p are the new task names for cor-

responding procedures, and t

are meta-

variables for time names such that t

≤ t

≤ t.

Time names of Π are 0, . . . , n. In practice, the last

time n is tested one by one from 1, 2, . . . until tr(∆)

has a causal model. In the example, n = 27 and a

computed plan of ∆ is:

move(2), grasp(bread), move(3),

putdown(bread), open(micro), grasp(bread),

putin(bread, micro), close(micro), turnon(micro),

wait, open(micro), takeout(bread, micro), check(bread),

putin(bread, micro), close(micro), turnon(micro),

wait, open(micro), takeout(bread, micro), check(bread),

putin(bread, micro), close(micro), turnon(micro),

wait, open(micro), takeout(bread, micro), check(bread).

At the beginning, the procedure P requires the ﬂuent

in(bread, micro) to be true, then based on the action

domain, the system computes a sub-plan to fulﬁll the

requirement:

move(2), grasp(bread), move(3), putdown(bread),

open(micro), grasp(bread), putin(bread, micro).

After that, the procedure turnon(micro);check(bread)

needs to be performed. However, both actions can-

not be executed directly as their preconditions are not

satisﬁed at the moment. Then the system ﬁlls the

missing details by respectively performing the action

close(micro) and the sequence of actions:

wait, open(micro), takeout(bread,micro).

In the loop procedure, as the environment has been

changed, the action putin(bread, micro) could be ex-

ecuted to accomplish the task Place(bread, micro).

Then the process continues until bread has been

heated in micro for three times. The above process

has been converted to the reasoning process of the

causal theory tr(∆) and executable plans of ∆ can be

extracted from corresponding causal models.

6 CONCLUSIONS

We present a formalization for structured task instruc-

tions which represents both functional and procedu-

ral knowledge of how to accomplish a task and pro-

vide an approach for integrating these structured in-

structions with robot’s built-in knowledge to com-

pute plans for tasks. The approach provides a basis

for allowing a robot to fulﬁll open-ended tasks using

knowledge obtained from web sources and/or through

natural human-robot communication. We have imple-

mented a prototype of the system and a case study is

reported on the effectiveness of the approach.

In our approach, both functional and procedural

knowledge are converted to causal rules, then they are

used in the uniform reasoning mechanism of causal

theories and worked together to compute plans for re-

quired tasks. Moreover, the approach could compute

executable plans when task instructions miss some in-

dispensable steps between two procedures. In par-

ticular, if an action cannot be executed in the pro-

cedure, the system would automatically add some

proper actions to satisfy preconditions of the action

which make the computed plan executable. Further-

more, the robot could add new knowledge during

the running process and use new knowledge to ful-

ﬁll tasks without changing its decision-making algo-

rithm.

ACKNOWLEDGEMENTS

This work is supported by the National Hi-Tech

Project of China under grant 2008AA01Z150 and

the Natural Science Foundation of China under grant

60745002 and 61175057, as well as the USTC Key

Direction Project, the USTC 985 Project, the Funda-

mental Research Funds for the Central Universities,

and the Youth Innovation Fund of USTC. Authors are

grateful to the other team members and the sponsors.

REFERENCES

Baral, C. (2003). Knowledge Representation, Reasoning

and Declarative Problem Solving. Cambridge Uni-

versity Press, New York, NY, USA.

Baral, C., Dzifcak, J., Gonzalez, M., and Zhou, J. (2011).

Using inverse lambda and generalization to trans-

late english to formal languages. Arxiv preprint

arXiv:1108.3843.

Beetz, M., Jain, D., M¨osenlechner, L., and Tenorth, M.

(2010). Towards performing everyday manipula-

tion activities. Robotics and Autonomous Systems,

58(9):1085–1095.

Brenner, M. and Nebel, B. (2009). Continual plan-

ning and acting in dynamic multiagent environ-

ments. Autonomous Agents and Multi-Agent Systems,

19(3):297–331.

Burgard, W., Cremers, A., Fox, D., H¨ahnel, D., Lakemeyer,

G., Schulz, D., Steiner, W., and Thrun, S. (1999).

Experiences with an interactive museum tour-guide

robot. Artiﬁcial Intelligence, 114(1-2):3–55.

Cantrell, R., Talamadupula, K., Schermerhorn, P., Benton,

J., Kambhampati, S., and Scheutz, M. (2012). Tell

me when and why to do it!: run-time planner model

FromStructuredTaskInstructionstoRobotTaskPlans

243

updates via natural language instruction. In Proceed-

ings of the 7th annual ACM/IEEE international con-

ference on Human-Robot Interaction (HRI-12), pages

471–478. ACM.

Chen, D. L. and Mooney, R. J. (2011). Learning to interpret

natural language navigation instructions from obser-

vations. In Proceedings of the 25th National Confer-

ence on Artiﬁcial Intelligence (AAAI-11), pages 859–

865.

Chen, X., Ji, J., Jiang, J., Jin, G., and Wang, F. (2010).

Developing high-level cognitive functions for service

robots. In Proceedings of the 9th International Con-

ference on Autonomous Agents and Multiagent Sys-

tems (AAMAS-10), pages 989–996.

Chen, X., Xie, J., Ji, J., and Sui, Z. (2012). Toward

open knowledge enabling for human-robot interac-

tion. Journal of Human-Robot Interaction, 1(2):100–

117.

Ferraris, P. (2007). A logic program characterization of

causal theories. In Proceedings of the 20th Inter-

national Joint Conference on Artiﬁcial Intelligence

(IJCAI-07), pages 366–371.

Galindo, C., Fern´andez-Madrigal, J., Gonz´alez, J., and

Safﬁotti, A. (2008). Robot task planning using se-

mantic maps. Robotics and Autonomous Systems,

56(11):955–966.

Gebser, M., Kaminski, R., Kaufmann, B., Ostrowski, M.,

Schaub, T., and Thiele, S. (2008). Engineering

an Incremental ASP Solver. In Proceedings of the

24th International Conference on Logic Programming

(ICLP-08), pages 190–205.

Ghallab, M., Howe, A., Christianson, D., McDermott, D.,

Ram, A., Veloso, M., Weld, D., and Wilkins, D.

(1998). Pddl—the planning domain deﬁnition lan-

guage. AIPS98 planning committee, 78(4):1–27.

Gupta, R. and Hennacy, K. (2005). Commonsense rea-

soning about task instructions. In Proceedings of

the AAAI-05 Workshop on modular construction of

human-like intelligence, pages 05–08.

Johnston, B. and Williams, M.-A. (2008). Comirit: Com-

monsense reasoning by integrating simulation and

logic. Frontiers in Artiﬁcial Intelligence and Appli-

cations, 171:200–211.

Kress-Gazit, H., Fainekos, G. E., and Pappas, G. J. (2007).

From structured english to robot motion. In Proceed-

ings of the 2007 IEEE/RSJ International Conference

on Intelligent Robots and Systems (IROS-07), pages

2717–2722. IEEE.

Levesque, H. J., Reiter, R., Lesperance, Y., Lin, F., and

Scherl, R. B. (1997). Golog: A logic programming

language for dynamic domains. The Journal of Logic

Programming, 31(1):59–83.

MacMahon, M., Stankiewicz, B., and Kuipers, B. (2006).

Walk the talk: connecting language, knowledge, and

action in route instructions. In Proceedings of the 21st

national conference on Artiﬁcial intelligence (AAAI-

06), pages 1475–1482. AAAI Press.

McCain, N. and Turner, H. (1997). Causal theories of action

and change. In Proceedings of the 14th National Con-

ference on Artiﬁcial Intelligence (AAAI-97), pages

460–465.

McDermott, D. (1992). Robot planning. AI magazine,

13(2):55–79.

Rybski, P., Yoon, K., Stolarz, J., and Veloso, M. (2007).

Interactive robot task training through dialog and

demonstration. In Proceedings of the 2nd ACM/IEEE

international conference on Human-robot interaction

(HRI-07), pages 49–56. ACM.

Son, T. C., Baral, C., and McIlraith, S. A. (2001). Plan-

ning with different forms of domain-dependent con-

trol knowledge – an answer set programming ap-

proach. In Proceedings of the 6th International Con-

ference on Logic Programming and Nonmonotonic

Reasoning (LPNMR-01), pages 226–239.

Tellex, S., Kollar, T., Dickerson, S., Walter, M., Banerjee,

A. G., Teller, S., and Roy, N. (2011). Understand-

ing natural language commands for robotic navigation

and mobile manipulation. In Proceedings of the 25th

National Conference on Artiﬁcial Intelligence (AAAI-

11).

Tenbrink, T., Ross, R. J., Thomas, K. E., Dethlefs, N.,

and Andonova, E. (2010). Route instructions in map-

based human–human and human–computer dialogue:

A comparative analysis. Journal of Visual Languages

& Computing, 21(5):292–309.

Tenorth, M., Nyga, D., and Beetz, M. (2010). Understand-

ing and executing instructions for everyday manipu-

lation tasks from the world wide web. In 2010 IEEE

International Conference on Robotics and Automation

(ICRA-10), pages 1486–1491. IEEE.

Thielscher, M. (2005). Flux: A logic programming method

for reasoning agents. Theory and Practice of Logic

Programming, 5(4-5):533–565.

Vogt, P. (2006). Language evolution and robotics: Issues on

symbol grounding. Artiﬁcial cognition systems, pages

145–173.

Xie, J., Chen, X., Ji, J., and Sui, Z. (2012). Multi-mode nat-

ural language processing for extracting open knowl-

edge. In Proceedings of the 2012 IEEE/WIC/ACM

International Conferences on Intelligent Agent Tech-

nology (IAT-12).

KEOD2013-InternationalConferenceonKnowledgeEngineeringandOntologyDevelopment

244