REMOTE HUMAN-ROBOT COOPERATION VIA INTERNET

USING WEBOS-BASED TOUCH INTERFACE

Igor Belousov

Technology Solutions Group, Hewlett-Packard, 16A, Bld. 3, Leningradskoe Shosse, Moscow, Russia

Keywords: Human-robot interface, Web-based robotics, Dynamic environments, HP webOS open mobile platform.

Abstract: We consider at this paper the complicated task when the robot manipulator is controlled remotely via the

World Wide Web and has to catch the moving object. The robot working environment is non-structured and

dynamic. Neither classical teleoperation nor pure robot programming could solve the task of grasping the

moving object in such a case. We used the shared autonomy approach to implement the capture task. The

operator plans the capture at high level, and the capture is implemented by the robot using vision system.

Human-Robot Interface is based on HP webOS Open Mobile Platform using HP Pre or HP TouchPad

(tablet computer) as the operator consoles. Using the touch-based control interface and real-time 3D models

of the remote robot and working environment make the grasping operation effective, reliable and simple.

1 INTRODUCTION

Achievements at Web-based robot control open new

promising application areas for robotic technologies.

Web-based tele-maintenance in the industrial

settings (Lou and Lee, 1999), tele-surgery

(Hannaford, 2008), remote robot control in

hazardous environments (Hamel, 2001), remote

education in robotics and mechatronics (Tzafestas,

2009) can be mentioned as the examples.

Internet itself is the ideal but challenging

environment for experimentation and testing wide

class of robotic systems such as teleoperated and

distributed systems (Nuno, Basanez and Prado,

2009), networked robots, haptic interfaces. Internet

provides natural and mature technology for remote

experimentations, but challenges the researchers by

such complications as limited bandwidth,

unpredictable and variable time delays, losses of

data packages, security problems.

To address the issues imposed by Internet we

have developed several VR-based methods for

effective robot teleoperation (Belousov, 2007). It

comprises: (1) an environment for off-line and on-

line remote robot programming via the Internet

(Belousov and Clapworthy, 2002), and (2) a Java3D-

based on-line dynamic virtual representation of the

remote robot and its working area (Tan, Clapworthy

and Belousov, 2004). These methods allow the time

delays inherent in IP networks to be suppressed, and

the operator's work to be simplified and accelerated.

Systems for control of the PUMA 560 and CRS

A465 manipulators and mobile robot Nomadic

XR4000 via the Internet have been developed.

Systems were successfully tested under real Internet

conditions from different locations in Russia,

England, France, South Korea.

Robot teleoperation via the Internet is difficult

task due to above mentioned reasons. But it rises to a

higher level of complexity when we consider the

interaction not only with fixed but also with moving

objects. Such dynamic environments are quite

difficult to cope with, but at the same time are the

most important from practical point of view because

the real world systems and environments are

dynamic, changing and unstructured.

Due to complexity of this task there are just few

articles addressing the problem. In (Kikuchi, Takeo

and Kosuge, 1998) the authors used bilateral

teleoperation subsystem to grasp a moving object.

They considered several conditions limiting the

scope of the approach – only 2D motion of the

object, permanent and slow speed of the object to

grasp, and constant and small time delays.

More general approach was presented in

(Belousov, Sazonov and Chebukov, 2005). The

Web-based robotic system presented was able to

grasp a fast moving object performing arbitrary 3D

motion in unstructured environment. Vision system

341

Belousov I..

REMOTE HUMAN-ROBOT COOPERATION VIA INTERNET USING WEBOS-BASED TOUCH INTERFACE.

DOI: 10.5220/0003649403410344

In Proceedings of the 8th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2011), pages 341-344

ISBN: 978-989-8425-75-1

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

and dynamic model of the object motion were used

to define and to predict the motion, and to perform

the grasping under supervision by human operator.

This approach allowed to develop sophisticated

algorithms for collision avoidance for the complex

robotic systems such as large space manipulators

(Belousov, Esteves, Laumond and Ferre, 2005) and

humanoid robots (Yoshida, Esteves, Belousov,

Laumond, Sakaguchi and Yokoi, 2008).

The main focus of the current work is

improvement of the system for Web-based robot

control in dynamic environment via the

enhancement of human-robot interaction using the

touch-based interface. It is based on HP webOS

Open Mobile Platform. The use of the touch-based

control interface and real-time 3D models of the

remote robot and working environment allows the

implementation of the grasping operation in natural

way. The operator plans the capture at high level

choosing for the robot the intermediate and final

goals (clicking on desired positions by a finger at the

touch screen with 3D scene model), and the capture

robot performs automatically.

Robot manipulator PUMA 560 was used at these

experiments (Fig. 1).

Figure 1: Robot Manipulator PUMA 560. Grasping the

Rod on Bifilar Suspension.

At the chapter 2 shared autonomy approach for

the task of grasping the moving object is presented.

Third chapter contains description of the system

architecture, data flows as well as touch-based

interface and HP webOS platform used. Experiments

undertaken are presented at the forth chapter.

Chapter 5 concludes the article and describes the

future system development.

2 SHARED AUTONOMY

APPROACH FOR

HUMAN-ROBOT

INTERACTION IN DYNAMIC

ENVIRONMENTS

We consider at this paper the complicated task when

the robot manipulator is controlled remotely via the

World Wide Web and has to catch the fast moving

object. The robot working environment is non-

structured and dynamic. Neither classical

teleoperation nor pure robot programming could

solve the task of grasping the moving object in such

a frame. We used the shared autonomy approach to

implement the capture task, i.e. the scenario when

actions of the human operator and the robot are

shared – operator (client side of the system)

implements the high-level planning and the robot

(remote or server side of the system) performs the

requested operations with high precision and

accuracy. Such a sharing allowed solving a

complicated task of capturing the fast objects which

move under the action of natural forces, and the

initial conditions of the motion can be arbitrary.

Maximum object speed in our experiments was over

1.5 m/sec.

The object used was a rod on a bifilar suspension

(Fig. 1). The upper ends of the threads are attached

to a fixed beam. The rod can perform complicated

free motion in 3 modes.

The vision system (VS) contains two TV

cameras placed above the scene and at the side to

ensure sufficient visual data to determine the

parameters of the real motion of the objects.

The VS determines the coordinates, in the image,

of some characteristic points on the object, relative

to the camera reference frame. The use of fast

image-processing algorithms allows up to 30 sets of

measurements per second to be obtained. The data

are collected during a time interval and are subjected

to statistical processing, taking into account the

mathematical model of the object motion. This gives

the initial conditions of the motion of the object and

allows the motion to be predicted for several

seconds ahead. The result of the prediction is the

precise time and the required position of the robot at

which the desired operation with the object should

be performed.

3 SYSTEM ARCHITECTURE

In the last few years Web technologies demonstrated

ICINCO 2011 - 8th International Conference on Informatics in Control, Automation and Robotics

342

the significant evolution. The most important trends

are standardisation based on HTML5 and CSS,

revolutionary improvements (run-time speed,

standartisation, cross-platform support) of

Javascript, move of the technology to mobile/

personal world with such amazing options as 3G/4G

fast Internet connection everywhere anytime and

touch-based interfaces on tablets and smartphones.

We decided to use all these advances and to develop

next generation of the system for Web-based robot

control in dynamic environment using these

technologies.

HP webOS – open Linux-based operating system

for mobile devices, - has been chosen (Allen, 2009).

It uses standard HTML5, CSS and Javascript for

applications development and provides good

opportunities for developers of remote teleoperated

robotics systems. Open operating systems with real-

time possibilities are extremely important for

developing robot control systems (Manchini and

Frontoni, 2009) and HP webOS is a good choice for

remote robot control from the mobile devices.

Other advantage of webOS is in its native

multitasking. The user experience is optimized for

launching and managing multiple applications at

once. HP webOS is designed around multitasking

and makes it simple to run background applications,

to switch between applications, and to easily handle

interruptions and events (Zammetti, 2009). These

features were used for the development of the client

part of the system.

The system contains 3 main parts – robot, server

and client. The robot part consists of the robot

manipulator and the robot controller. The server part

contains the server computer and TV cameras. We

have used Palm Pre smartphones and HP TouchPad

tablet as the client control devices. All the client

software is realised with open technologies

javascript and OpenGL ES. We used HP webOS

SDK 2.1 (Pre 2), and SDK 3.0 for TouchPad.

The robot controller was connected to the server

computer via RS232 serial interface.

Communication between the server and client parts

was performed using TCP/IP packages.

The software of the robot part of the system

provides communication with the server part (bi-

directional data exchange) and control of the robot

itself. It has been realised using VAL-type language.

The server part of the system contains the

software modules for the data exchange with both

the robot and the client parts, and a module for TV-

image processing.

The software of the client part consists of

modules for communication with the server, a

module for robot control, and the modules for

visualisation of the robot and working environment

– 3D graphic representation and TV images.

Important part of the client side is a human-robot

interface. The functional description is presented at

Chapter 4, but here it is important to emphasize that

it is build using HP webOS UI and provides rich

touch-based user experience in a simple and logical

way. It supports multi-touch and multiple gestures

such as tapping, flicking, swipe, dragging, scrolling

and others. This makes operator’s work comfortable

and natural.

4 EXPERIMENTS

We tested the above algorithms and control

environment to grasp the rod by the PUMA

manipulator when operator was located far away

from the robot and controlled it via the low-

bandwidth Internet connection.

At the start of the control program, the rod is

placed in the equilibrium position. A remote

operator can define the initial motion of the rod by

choosing in the on-line webOS-based Virtual

Environment (VE) any point on the rod surface

picking the point with the finger; robot hits that

point with a random speed to provide random

character of the oscillations. Operator observes the

rod motion in the VE and chooses in the VE the

desired grasping point and instant to begin the

grasping operation. Chosen values are transmitted to

the sever (i.e. robot) site and control program

automatically: 1) predicts the capture instant and

position of the manipulator gripper for that time, 2)

controls the capture, 3) checks the capture

implementation, and 4) decides to repeat capture in

the event of failure. Operator continues to observe

the rod oscillations and grasping process in VE and

can stop the control program and re-plan grasping

scenario if needed.

Operator used HP Pre 2/3 webOS-based

smartphones and HP TouchPad to control the remote

robot and the control interface is presented at Fig. 2.

Window with 3D models of the robot and the

environment is located at the right-hand side of the

picture. Since only several numbers should be

transmitted from the server to visualise the rod,

scene redraw was performed almost at real time

scale even for a low bandwidth connection.

Components of the robot control interface are placed

at the left-hand part of the picture.

System has been tested from different remote

REMOTE HUMAN-ROBOT COOPERATION VIA INTERNET USING WEBOS-BASED TOUCH INTERFACE

343

locations and proved to be an efficient and reliable –

grasping was performed in 100% cases.

Figure 2: Operator’s Control Interface on HP TouchPad as

the control device.

5 CONCLUSIONS

Complicated task of grasping the moving object by a

robot manipulator when controlling the robot via the

Internet has been solved. Use of the shared

autonomy approach (when operator forms initial

motion of the rod and plans grasping operation, and

robot performs final operation automatically)

allowed to suppress communication delay, and also

to use in appropriate proportion operator’s skills in

planning high-level operations and robot accuracy.

HP webOS-based HP Pre 2/Pre 3 smartphones

and TouchPad tablet were used as the control

devices by the operator for remote robot control.

And the operator used natural gesture manipulations

on the live 3D model of the scene for high-level

planning of the grasp operations.

Methods developed could be applied to the wide

class of the remotely controlled systems with the

delays in control loop.

Future work will be focused on adding the

modules for automatic obstacle avoidance, remote

programming of the robot in touch human-operator

interface and use of WebGL language (3D extension

for Javascript, http://www.khronos.org/webgl/) for

3D visualisation of the robot and its working

environment at HP webOS devices.

REFERENCES

Lou, R., Lee, J., 1999. Tele-Control of Rapid Prototyping

Machine Via Internet for Automated Tele

Manufacturing. In IEEE International Conference on

Robotics & Automation ICRA’99, Detroit (USA), May

1999, pp.2203-2208.

Kikuchi, J., Takeo, K., Kosuge, K. 1998. Teleoperation

system via computer network for dynamic

environment. In IEEE International Conference on

Robotics & Automation ICRA’98, Leuven (Belgium),

May 1998, pp. 3534-3539.

Bakers, P., Tso, K., 2000. Internet-based Operations for

the Mars Polar Lander Mission, In IEEE International

Conference on Robotics & Automation ICRA’2000,

San Francisco (USA), April 2000, pp.2025-2032.

Hamel, W., 2001. Observations concerning Internet-based

teleoperations for hazardous environments. In IEEE

International Conference on Robotics & Automation

ICRA'2001, Seoul (Korea), May 2001, pp. 638-643.

Hannaford, B., 2008. Telesurgery via Internet, Workshop

“New Vistas and Challenges in Telerobotics”, In IEEE

International Conference on Robotics and

Automation, Pasadena (USA), May 2008.

Tzafestas, S., 2009, Web-based Control and Robotics

Education, Springer, 296 p.

Nuno, E., Basanez, L., Prada, M., 2009. Asymptotic

Stability of Teleoperators With Variable Time-Delays,

In IEEE International Conference on Robotics and

Automation, Kobe (Japan), May 2009.

Manchini, A., Frontoni, E., 2009. RoboBuntu: A Linux

Distribution for Mobile Robotics. In IEEE

International Conference on Robotics and

Automation, Kobe (Japan), May 2009.

Belousov, I., Clapworthy, G., 2002. Remote programming

and Java3D visualisation for Internet robotics, SPIE’s

International Technical Group Newsletter, Vol. 11,

No. 1, February 2002.

Tan, J., Clapworthy, G., Belousov, I., 2004. The

Integration of a Virtual Environment and 3D

Modelling Tools in a Networked Robot System,

International Journal of Image and Graphics, World

Scientific Publishing Company, Sept. 2004, pp. 1-20.

Belousov, I., Sazonov, V., Chebukov, S., 2005. Web-

based Teleoperation of the Robot Interacting with Fast

Moving Objects, In IEEE International Conference on

Robotics and Automation ICRA'05, Barcelona (Spain),

April 2005.

Belousov, I., Esteves, C., Laumond, J.-P., Ferre, E. 2005.

Motion planning for the large space manipulators with

complicated dynamics. In IEEE/RSJ International

Conference on Intelligent Robots and Systems

IROS’2005, Edmonton (Canada), August 2005, pp.

3713-3719.

Belousov, I., 2007. Sidebar – Java3D for Web-based

Robot Control, Invited Article in “Advances in

Telerobotics”, Volume 30, 2007, Springer-Verlag.

Yoshida, E., Esteves, C., Belousov, I., Laumond, J.-P.,

Sakaguchi, T., Yokoi, R., 2008. Planning 3D

Collision-Free Dynamic Robotic Motion Through

Iterative Reshaping, IEEE Transaction on Robotics,

Vol.24, N°5, October, 2008, pp.1186-1198.

Allen, M., 2009, Palm webOS, O’Reilly, 2009, 432 p.

Zammetti, F., 2009. Practical Palm Pre webOS Projects,

Apress, 2009, 400 p.

ICINCO 2011 - 8th International Conference on Informatics in Control, Automation and Robotics

344