PLATFORM-BASED TELEOPERATION CONTROL OF SYMBIOTIC

HUMAN-ROBOT SYSTEM

Tao Zhang, Vuthichai Ampornaramveth, Md. Hasanuzzaman, Pattara Kiatisevi, Haruki Ueno

Intelligent Systems Research Division, National Institute of Informatics

2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo 101-8430, Japan

Keywords:

Platform-Based Teleoperation Control, Symbiotic Human-Robot System, SPAK, Robovie, AIBO.

Abstract:

This paper presents a platform-based teleoperation control approach of symbiotic human-robot system. With

frame-based knowledge representation, features of robots, human-robot interface and cooperative operation

of symbiotic human-robot system are deﬁned in the Software Platform of Agents and Knowledge Manage-

ment (SPAK). By means of this software platform, human can communicate with robots using human-robot

interface. Cooperative operation of multiple robots can be implemented by teleoperation control through wire-

less network. In this paper, platform-based teleoperation control of an actual symbiotic human-robot system

comprised of human, humanoid robot (Robovie) and entertainment robot (AIBO) is implemented and the

experimental results demonstrate its effectiveness.

1 INTRODUCTION

With the development of robotic techniques, symbio-

sis of human and robots is proposed for construct-

ing high-intelligent, human-friendly welfare society

(H. Ueno, 2002). In such a symbiotic human-robot

system, human is just a ordinary member be able

to acquire services from robots according to his re-

quests. Robots can understand human intention and

act their behaviors for human. In addition, many types

of robots with various functions in this system can co-

operatively work for a speciﬁc task. They are good

servers of human and also friendly partners of human.

Therefore, it is an attractive and promising topic to

construct a symbiotic human-robot system.

However, a symbiotic human-robot system is com-

prised of many different types of robots and should

integrate many kinds of techniques for implement-

ing various complex tasks. In addition, as one of

essential conditions, activity of symbiotic human-

robot system should be conducted under wireless

network. Therefore, many conventional control ap-

proaches only in views of a speciﬁc robot are not

ﬁt for symbiotic human-robot system comprised of

many different robots (K.J. Schilling, 1999). In this

research, a platform-based teleoperation control ap-

proach is proposed in order to implement complex

control of symbiotic human-robot system.

In order to implement platform-based teleoperation

control of symbiotic human-robot system, regarding

various features of robots, human-robot interaction

as well as cooperative operation, all of them are de-

ﬁned in the software platform by frame-based knowl-

edge representation. Since the teleoperation control

is completely depended on the deﬁnition of symbi-

otic human-robot system in software platform, soft-

ware platform should comprise of many functions.

In our laboratory, a software platform, called Soft-

ware Platform for Agents and Knowledge Manage-

ment (SPAK) (V. Ampornaramveth, 2003), is devel-

oped recently. This is a powerful platform and can

provide great supports to the teleoperation control of

symbiotic human-robot system.

The remainder of this paper is organized as follows.

In section II, a symbiotic human-robot system is de-

ﬁned in SPAK by frame-based knowledge represen-

tation, including features of robots, human-robot in-

terface and cooperative operation. Through wireless

network, symbiotic human-robot system is controlled

according to the deﬁnition of cooperative operation in

SPAK in section III. Section IV introduces an actual

symbiotic human-robot system comprised of human,

humanoid robot (Robovie) and entertainment robot

(AIBO) and its experimental results demonstrate the

effectiveness of this method.

312

Zhang T., Ampornaramveth V., Hasanuzzaman M., Kiatisevi P. and Ueno H. (2004).

PLATFORM-BASED TELEOPERATION CONTROL OF SYMBIOTIC HUMAN-ROBOT SYSTEM.

In Proceedings of the First International Conference on Informatics in Control, Automation and Robotics, pages 312-317

DOI: 10.5220/0001133803120317

 SciTePress

2 DEFINITION OF SYMBIOTIC

HUMAN-ROBOT SYSTEM IN

SOFTWARE PLATFORM

In a symbiotic human-robot system there are many

different types of robots. All parts of this system

are connected with a wireless network. With human-

robot interface developed by many kinds of tech-

niques, such as speech recognition, image analysis,

etc., robots can communicate with human and under-

stand human requests. Under the control of SPAK,

robots can act their behaviors themselves or perform

cooperative operation among them.

SPAK is a frame-based knowledge engineering en-

vironment (V. Ampornaramveth, 2003). It provides

a central module, which acts as blackboard, knowl-

edge processing brain, memory, and do the judgment,

task planning and execution. It also provides soft-

ware tools necessary for integration of various exist-

ing modules over a TCP/IP network. The features of

SPAK are ”platform-independent” as existing robots

and software modules often rely on different plat-

forms or operation systems, ”network-aware” as the

modules must interact on a network, supporting ”soft-

ware agent” and being ”user friendly”. SPAK is tar-

geted to be the platform on which a group of coordi-

native robots (or their agents) operate on top of frame

knowledge. As illustrated in Fig.1, SPAK consists of

the following software components:

• GUI Interface: A user-friendly graphical interface

to the internal knowledge manager and the infer-

ence engines. It provides the users direct access to

the frame-based knowledge.

• Knowledge Database and Knowledge Manager:

This is the SPAK core module which maintains

the frame systems as Java class hierarchy, and per-

forms knowledge conversion to/from XML format.

• Inference Engines: Verify and process informa-

tion from external modules which may result in in-

stantiation or destruction of frame instances in the

knowledge manager, and execution of predeﬁned

actions.

• JavaScript Interpreter: Interprets JavaScript code

which is used for deﬁning condition and procedural

slots in a frame. It also provides access to a rich set

of standard Java class libraries that can be used for

customizing SPAK to a speciﬁc application.

• Base Class for Software Agent: Provide basic func-

tionality for developing software agents that reside

on networked robots.

• Network Gateway: This is a daemon program al-

lowing networked software agents to access knowl-

edge stored in SPAK. All SPAK network trafﬁcs are

processed here.

Figure 1: Structure of SPAK.

In order to implement teleoperation control by

SPAK, the symbiotic human-robot system should be

deﬁned in advance. Its deﬁnition is based on the

frame-based knowledge representation. It is well

known that frame representation systems are cur-

rently the primary technology used for large-scale

knowledge representation in Artiﬁcial Intelligent (AI)

(D. Koller, 1998). A frame is a data-structure for rep-

resenting a stereotyped situation (M. Minsky, 1974).

Attached to each frame are several kinds of informa-

tion. Collections of related frames are linked together

into frame-systems. The structure of a frame is con-

sisted of several items, such as name, type, A-kind-

of, Descendants, slots, etc. (G. Tairyou, 1998). As

the element of a frame, each slot has the following

elements, such as name, type, values, conditions, etc.

Using frames and their slots, features of various

robots in a symbiotic human-robot system can be de-

ﬁned, which are classiﬁed by their types, spatial po-

sition, functions, etc. From these features, we can fa-

miliar with each type of robot and further ﬁnd out the

robots with identical features. Human-robot interface

is also deﬁned, including frames for speech, recogni-

tion, etc. In addition, the relationship among robots

can be deﬁned. The types of relationship among

robots include synchronization, succession, restric-

tion, etc. As we are deﬁning the coordinative con-

trol of symbiotic human-robot system for perform-

ing a speciﬁc task, the knowledge on the relationship

among robots will provide sufﬁcient information and

simplify the plan procedure. Besides, many possible

behaviors are deﬁned, which can be implemented by

robots, such as walking, shaking hands, etc. They

provide great support to the activities of symbiotic

human-robot system.

In SPAK, the frame-based knowledge representa-

tion of a symbiotic human-robot system is imple-

mented by means of XML format, as illustrated by

Table.1. XML is a markup language for documents

PLATFORM-BASED TELEOPERATION CONTROL OF SYMBIOTIC HUMAN-ROBOT SYSTEM

313

containing structured information. With XML format,

frame structure as well as its contents written by slots

can be deﬁned easily. Particularly, the frame system

can be illustrated in the SPAK Graphic User Interface

(GUI). In the frame system, all frames are connected

by use of ISA relations. The ISA relation means that

there has the class and sub-class relation between the

upper frame and the lower frame. The lowest frame is

an instance of the upper frame. Besides, correspond-

ing to XML ﬁle, there is an interpreter to translate

XML speciﬁcation into relative commands.

Table 1: XML format in SPAK

< FRAME >

< NAME >c

< / NAME >

< ISA >c

< / ISA >

< ISINSTANCE >...

< / ISINSTANCE >

< SLOTLIST >

< SLOT >c

< / SLOT >

< SLOT >c

< / SLOT >

< / SLOTLIST >

< / FRAME >

The meaning of this format can be explained cor-

responding to the deﬁnition of a frame. Between

<FRAME> and < /FRAME> deﬁnes a frame.

NAME refers to the frame name. ISA refers to

the item of ”A-kind-of”. ISINSTANCE refers to

the frame type. Between <SLOTLIST> and <

/SLOTLIST> deﬁnes slots. Each SLOT gives the

contents of a slot. The symbol ”c” means here should

have some contents. With this XML format, a frame

and its slots can be described in detail. Therefore,

a symbiotic human-robot system deﬁned by framed-

based knowledge representation can be implemented

in SPAK by means of XML format.

3 TELEOPERATION CONTROL

OF SYMBIOTIC HUMAN-

ROBOT SYSTEM

With the deﬁnition of symbiotic human-robot system

in SPAK, teleoperation control can be implemented.

There are three important behaviors which should be

conducted in the symbiotic human-robot system by

teleoperation control for most of tasks. The ﬁrst is to

realize interaction between human and robots in or-

der to get human requests. In SPAK integrates several

techniques for implementing human-robot interface,

such as face detection, image recognition, etc. Con-

cerning face detection, by use of ”eyes” of robot, in-

stalled by cameras, robot can get the image of human

face. With the program of face detection storing in the

center computer of robot, robot can recognize human

by what it looks. Of course, if it is the ﬁrst time that

robot looks this person, robot can store the enough in-

formation about this person’s face in his database by

its learning program. As robot looks this person’s face

again, he can recognize him at once. Thus it sends the

recognition results to SPAK. This result can trigger

other actions of robots.

Another technique that robot can use for human-

robot interface is by means of speech with human.

Robot will ﬁrstly store the information on the speech

of human. As robot hears the similar speech again,

he can recognize this person at once. Regarding

these two techniques, SPAK includes the deﬁnition of

human-robot interface. With this interface, many in-

dependent programs for performing various functions

in robots, such as image capture, speaking, etc. can

coordinative work to implement interaction between

robot and human.

The second behavior is to implement cooperative

operation. For cooperative operation, ﬁrstly, it needs

to trigger the activity of symbiotic human-robot sys-

tem. The simple way is to give a simple command

to indicate the beginning of the activity. But actually,

there are also many other kinds of ways to trigger ac-

tivity. One of the most important ways is based on

human-robot interaction. Next, the plan of activity of

symbiotic human-robot system is necessary for coop-

erative operation. The plan of activity should obey the

relations of all robots. The synchronization relation

means that several robots can be operated together at

the same time for a speciﬁc task. Their control in-

structions are generated referring to a same time axes.

The command in each time sampling period is possi-

ble for any robots. The succession relation means that

one robot should start its action after another robot

ﬁnishes its action. The restriction relation means that

as one robot is performing a certain action, another

robot can not be acting any actions at the same time.

The action of one robot will interrupt the action of

another robot. If we obey these three relations, even

a complex task could be undertaken by coordinative

control of symbiotic human-robot system.

In SPAK, the above three relations are imple-

mented by the following patterns. Since features of

robots are deﬁned by frames and in SPAK the infer-

ence engines for doing forward and backward chain-

ing are deﬁned, the relations can be implemented by

inference engines. The feedforward chaining is usu-

ally used when a new fact is added to the knowledge

ICINCO 2004 - ROBOTICS AND AUTOMATION

314

base and we want to generate its consequences, which

may add new other facts, and trigger further infer-

ences. The backward chaining starts with something

we want to prove, ﬁnd implication facts that would

allow us to conclude it. It is used for ﬁnding all an-

swers to a question posed to the knowledge base. For

these three relations, if the required argument is ob-

tained, a new fact will be generated. Corresponding

to this new fact, if there has a synchronization rela-

tion, several robots would perform a same task at the

same time. If there deﬁnes an action of a robot trig-

gered by this fact, the succession relation will be then

performed. Similarly, a restriction relation will cause

the robot stopping its actions. When we use backward

chaining, for a task there will have many actions from

various robots if they have cooperative relations.

The third behavior is the control of activity of sym-

biotic human-robot system. Since different robots

have different driving programs, the control of ac-

tivity of symbiotic human-robot system is actually

through the agents deﬁned in SPAK. The agents will

convert the control instructions generated in SPAK to

the robots. Basically, SPAK will send commands di-

rectly to each robot according to its plan of activity of

symbiotic human-robot system. Regarding the con-

trol of symbiotic human-robot system, feedforward

control to robots is not enough to perform the task.

Signal feedback from robots is deﬁnitely necessary to

evaluate the activity of robots as well as instruct the

next actions of robots. There are two ways to get the

feedback signals on the activity of symbiotic human-

robot system. In the environment where human and

robots are staying, we setup several cameras as the

sensors to capture the status of robots. Based on the

human’s judgment on the actions of robots, SPAK can

adjust its control signals or generate new tasks. An-

other way to get the feedback signals is by the robots

themselves. As robots ﬁnished one of their actions,

they should send back a response corresponding to

their actions. Moreover, since there are many sen-

sors in robot bodies, they could also send some sig-

nals detected by these sensors to SPAK, which could

be helpful for SPAK to know the status of the activity

of symbiotic human-robot system.

Finally, the teleoperation control of symbiotic

human-robot system is carried out as the following

process. In this system, all components are staying

at the state of connection at any time. The actua-

tion of behaviors of this system is from the beginning

of human-robot interaction. As robot gets human re-

quests through the interaction, it will send relative in-

formation to SPAK. Then, SPAK will plan the action

of symbiotic human-robot system according to human

request and generate instruction to robots. Robots

will offer responses to human. These robots will con-

duct relative behaviors independently or through co-

operative operations for human controlled by SPAK.

4 EXPERIMENT

An actual symbiotic human-robot system is con-

structed, which is comprised of human, humanoid

robot (Robovie), entertainment robot (AIBO), etc., as

illustrated by Fig.2. Robovie is developed by Ad-

vanced Telecommunications Research Institute Inter-

national (ATR) of Japan. It is a kind of autonomous

communication robots that can communicate and in-

teract with human in our daily environment. It has

11 degrees of freedom and sensors such as an omni-

directional camera, pan-tilt zooming cameras, micro-

phone, ultrasonic range sensors, tactile sensors, etc. It

also can move by its wheels. Robovie also provides

many demo programs for speech, camera vision, mo-

tion, etc. Based on the functions of Robovie, we have

developed several programs for monitoring the sta-

tus of all sensors and sending them to SPAK, deﬁn-

ing many actions of Robovie as well as instructing

Robovie to talk with human. Robovie can be used in

the ﬁeld of education, nursing, medical treatment, etc.

Figure 2: An actual symbiotic human-robot system.

AIBO is a kind of entertainment robots and devel-

oped by Sony Co. of Japan. It can provide high de-

gree of autonomous behavior and functionality. In our

experimental system, we use AIBO ESP-220, which

is able to walk on four legs. It has a total of 16 ac-

tuators throughout its body to control its movements,

and 19 lights on its head, tail, and elsewhere to ex-

press emotions like happiness or anger and reactions

to its environment.

As shown in Fig.3, a frame hierarchy written by

XML format is deﬁned in SPAK. All frames are con-

nected by their ISA relations. For ”Robot” frame,

there includes ”Robovie” and ”AIBO”. Related with

Robovie, there are many frames about Robovie’s

components (”Mouth”, ”Motor”, etc), Robovie’s

sensors and Robovie’s behaviors (”FaceDetection”,

PLATFORM-BASED TELEOPERATION CONTROL OF SYMBIOTIC HUMAN-ROBOT SYSTEM

315

Figure 3: Frame hierarchy of symbiotic human-robot system.

”Greet”, etc.). The Robovie’s behaviors can be

used for human-robot interface. For AIBO, there

are many frames on its actions. Some of them are

AIBO’s atomic actions (”AIBOStand”, ”AIBOSit”,

etc.). Others of them are about AIBO’s behav-

iors comprised of several atomic actions. Con-

cerning the users, there includes frames about new

users (”NewUser”) and known users (”KnownUser”).

Since many frames for symbiotic human-robot sys-

tem have been deﬁned, if relative conditions deﬁned

in the frame are satisﬁed, this frame will be actuated

and commands given in this frame will be performed.

Fig.4 is the slot-editing table showing slots belonging

to a frame of AIBO. In this frame, we can know that

if three arguments are given by relative frames, the

command deﬁned in the slot ”onInstantiate” will be

performed. With this system, coordinative control of

symbiotic human-robot system is performed by tele-

operation operation through SPAK.

The process of cooperative operation can be de-

scribed as below.

• With the human-robot interface installed in

Robovie, Robovie can recognize human and send

information about human to SPAK, such as user’s

name;

• With the results of human-robot interaction, SPAK

instructs Robovie and AIBO to start activity;

• As deﬁned beforehand on the activity of symbi-

otic human-robot system, Robovie informs human

about AIBO actions ﬁrstly;

• AIBO moves.

• As SPAK gets the response from AIBO that indi-

cates the end of AIBO movement, SPAK will in-

struct next action for both Robovie and AIBO.

Fig.5 shows the experimental environment. The ex-

perimental scenario is given in the next page. As do-

ing this experiment, all instructions are generated by

SPAK and sent to each robot. After each action, each

robot should inform its status to SPAK. According to

the status, SPAK will send the next command to each

robot according to the designed plan. With the control

of SPAK, the whole system performs the task condi-

tionally and automatically.

ICINCO 2004 - ROBOTICS AND AUTOMATION

316

Figure 4: Slot-editing table showing slots belonging to a frame of AIBO.

Figure 5: Experimental environment by use of symbiotic

human-robot system.

Robovie: (Robovie looks at the person’s face

and try to recognize it.)

Robovie: Hi, how are you! I have never seen

you before. What is your name?

(Robovie shakes it hand a little bit.)

Human: (A person types his name XXX into

the computer of Robovie.)

Robovie: Hi, XXX. How are you going today?

AIBO will play with you.

(Robovie shakes its both hands with

person while it is speaking with the

person.)

Robovie: AIBO is standing up now.

AINO: (AIBO stands up and sends back the

response to SPAK after its

movement.)

Robovie: AIBO is walking up now.

AINO: (AIBO walks a little bit and sends

back the response to SPAK after its

movement.)

Robovie: AIBO is turning to the left with 90

degree now.

AINO: (AIBO turns to the left with 90

degree and sends back the response to

SPAK after its movement.)

...

5 CONCLUSIONS

A new platform-based teleoperation control approach

of a symbiotic human-robot system is proposed. In

this method, features of various robots, human-robot

interface and cooperative operation can be deﬁned

by frame-based knowledge representation and imple-

mented in SPAK by XML format. By use of this

method, an actual symbiotic human-robot system was

constructed and its experimental results demonstrate

the effectiveness of this method. In the further re-

search, we will apply this system for performing more

complex tasks, especially using it for welfare enter-

prise, such as assisting the elder living at home. Since

this symbiotic human-robot system is open to any

kinds of robots as well as human-robot interfaces, it

can be extended widely for many kinds of applica-

tions.

REFERENCES

D. Koller, A. P. (1998). Probabilistic frame-based systems.

In Proc. of the 15th National Conference on AI, pages

580–587.

Minsky, M. (1974). A framework for representing knowl-

edge. MIT-AI Laboraroty Memo 306.

Tairyou, G. (1998). Development of frame-based knowl-

edge engineering environment. Master’s thesis, Tokyo

Denki University.

Schilling, K. J. and Roth, H. (1999). Control interfaces for

teleoperated mobile robots. Proceedings of 7th IEEE

International Conference on Emerging Technologies

and Factory Automation, 18-21.

Ueno, H. (2002). A knowledge-based information mod-

eling for autonomous humanoid service robot. IE-

ICE Transactions on Information and systems, E85-

D(4):657–665.

V. Ampornaramveth, P. Kiatisevi, H. U. (2003). Toward

a software platform for knowledge management in

human-robot environment. Technical Report of IE-

ICE, 103(83):15–20.

PLATFORM-BASED TELEOPERATION CONTROL OF SYMBIOTIC HUMAN-ROBOT SYSTEM

317