MODEL BASED CONTINUAL PLANNING AND CONTROL

FRAMEWORK FOR ASSISTIVE ROBOTS

A. Anier and J. Vain

Tallinn University of Technology, Ehitajate tee 5, 19086 Tallinn, Estonia

Keywords:

Assistive robotics, Model based control, Continual planning, Cognitive architecture.

Abstract:

The paper presents a model based robot planning and control framework for human assistive robots of medical

domain, namely for Scrub Nurse Robots. We focus on endoscopic surgery as one of the most relevant surgery

type for the use of robotic assistants. We demonstrate that our framework provides means for seamless inte-

gration of sensor data capture, cognitive functions, model based continual planning and direct actuator control.

The novel component of the architecture is a distributed continual planning system implemented based on the

Uppaal model checking and testing tool suite. The distributed and the contract-based modular architecture of

proposed framework enables ﬂexible online reconﬁguration and adaptability to various applications, but also

safe installation of new software components on-the-ﬂy.

1 INTRODUCTION

The assistive robotics sets high standards to cogni-

tive capabilities, autonomy and movement precision

of robots. Intuitively, it means understanding hu-

man intention and adequate reaction to it. Techni-

cally, it means human-in-the-loopcollaborativeaction

control, fusion of various sensor information, high

accuracy actuation and reliable software implemen-

tation. Safety issues of action and trajectory plan-

ning become critical in the conditions where the robot

shares user’s working envelope and the contact due

to the expected physical interaction for transfer of

objects is frequent. This paper presents a new soft-

ware integration framework for a Scrub Nurse Robot

(SNR)(Miyawaki et al., 2005) focusing on distributed

model based continual planning and control issues.

The goal of a SNR is to learn the interactions be-

tween a surgeon and a scrub nurse during a laparo-

scopic surgery and to replace the (human) nurse on

demand. One of the important aspects of incorporat-

ing the SNR in the collaborative action, e.g. when the

human scrub nurse is occupied with other tasks avoid-

ing the need for the surgeon to re-adapt to the robot

partner and preserving the “original feel”. An exam-

ple of surgical scene with scrub nurse waiting for right

moment of handing over an instrument to surgeon is

depicted in Fig. 1.

A scrub nurse must hand a surgical instrument to

a surgeon as soon as it is requested. If the scrub nurse

Figure 1: SNR intraoperative scene.(Miyawaki et al., 2005)

has to spend time searching for the instrument after a

request, the procedure is interrupted, valuable time is

lost and an unnecessary burden is placed on the sur-

geon. That possibly reduces the quality and effective-

ness of the operation. To avoid such delays the scrub

nurse must be fully attentive to the activity in the op-

erative ﬁeld and anticipate accurately what a surgeon

will need. For this to be possible the scrub nurse not

only needs to know the surgical procedure as well as

the surgeon does, but must also be highly disciplined.

The “ideal” scrub nurse (if one exists) is able to pass a

surgeon whatever is needed without any verbal order

at the moment that the surgeon’s hand is extended to

receive it.

The goal of SNR software project is to develop a

human-adaptiveSNR capable of adapting to surgeons

403

Anier A. and Vain J..

MODEL BASED CONTINUAL PLANNING AND CONTROL FRAMEWORK FOR ASSISTIVE ROBOTS.

DOI: 10.5220/0003827104030406

In Proceedings of the 2nd International Conference on Pervasive Embedded Computing and Communication Systems (PECCS-2012), pages 403-406

ISBN: 978-989-8565-00-6

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

with various levels of skill and experience, and even

to different personalities (and moods). In other words,

the SNR should be able to function as an “ideal” scrub

nurse. To attain this ideal, highly developed cognitive

faculties such as machine vision and speech recogni-

tion as well as adaptive robotic arm path planning and

targeting are required.

In conventional surgical operations a scrub nurse

frequently has to handle an array of different instru-

ments. It is very difﬁcult to make the SNR adaptive to

such busy operations. Therefore, the SNR prototype

has been designed for endoscopic surgery which does

not need many types of surgical instruments. The

adaptivity of the SNR requires unsupervised learning

by observing skilled nurses’ actions and behavior dur-

ing surgical operations.

On-line recognition and anticipating surgeon’s

motions while operating is essential to classify which

motions are common to all surgeons and which are

speciﬁc to individuals. This, in turn, will aid in an-

ticipating a surgeon’s needs and in adapting to the

changes of procedure. On the other hand, the results

of the investigation of intraoperative behavior have to

be abstracted and memorized in the form of mathe-

matical and/or formal models in order to reproduce

the variety of motion trajectories that can be expected

from various combinations of surgical procedures and

varying external factors. The model of a nurse’s be-

haviors as he or she reacts to other surgical staff (sur-

geon, assistant and others) serves as a high-level be-

havior speciﬁcation for the SNR action planning.

The SNR’s control architecture depicted in Fig.

2 comprises following components: MotionAnalysis

Hawk position tracking system that is capable of mea-

suring the position-tracking markers coordinates with

precision more than 1 mm with sampling rate up to

200 frames per second. The surgeon’s hand move-

ment sampling data is passed to gesture recognition

module that uses multiple recognition methods and

voting automata(Vain et al., 2009)for detection of sur-

geon’s current motion.The identiﬁed motion and its

parameters are inputs for reactive motion planning

that compares the observed movement of surgeon’s

hand with that of predicted by surgeon’s behavior

model and surgery scenario model, and makes the cor-

rection of the current model state, if necessary. By

the corrected state information and surgery scenario

model the next SNR action is planned and the result-

ing control parameters are transferred to the motion

control unit. The information about surgeon’s pos-

sible reactions predicted by the surgeon model is re-

turned to the motion recognition module to restrict the

decisions space when new movement is being recog-

nized.

Figure 2: Control loop outline.

2 SOFTWARE ARCHITECTURE

The control architecture described above is imple-

mented on the software framework written in C/C++

and Java and is designed around the Uppaal tool suite.

2.1 Data Acquisition

SNR doesn’t have integrated vision but makes use of

an external one. MotionAnalysis Hawk near-infrared

active 3D measurement system (3DMS) is used for

visual feedback. 3DMS is not the only source of in-

formation. There are various sensors to monitor the

state of the robot and peripheral interfaces that con-

tribute to the overall awareness of the environment.

Software architecture depicted in Fig. 3 uniﬁes

3DMS data with other data acquisition sources for

easier integration with upper architecture tiers. This

includes capturing RFID information about the in-

strument positions in use. All the instruments are

equipped with ceramic RFID tags that allow easy de-

tection when the instrument is inserted to or removed

from trocar cannula. Video image from the laparo-

scopic camera is used for anticipation of surgeon’s

motions, but it is not discussed in this paper. Last

but not least, the middle-ware enables automated ex-

ecution of capture and training sessions for algorithm

evaluation, optimization and testing.

2.2 Data Analysis and Cognitive

Functions

The robot control framework provides a common

platform for integration of data acquisition and cog-

PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems

404

Computer mouse3DMS cameras

TRC

Socket

EvaComm2 SDK

Mouse Tracker

MotionAnalysis EVaRT

Visualization

Configuration

jEvart

Object definition subsystem

Rapid Miner

Uppaal TRON (DTRON)

Voting automata

TA automataTA generator

Spread Proxy

Filtering

Motion rec.

Spread op.

EVaRT op.

Filtering

Motion rec.

Spread op

EVaRT op.

Filtering

Motion rec.

Spread op

EVaRT op.

Spread

SNR

Figure 3: Architecture model.

nitive functions.

Data analysis and cognitive functions are imple-

mented by means of data mining toolkit Rapid Miner.

It includes hundreds of algorithms ranging from ﬁlter-

ing to machine learning packaged into an integrated

development environment. Rapid Miner is inspired

by WEKA machine learning toolkit(Hall et al., 2009)

but with extensive data visualization and analysis au-

tomation tools.

To make the Rapid Miner ﬁt the SNR overall

control architecture some custom plug-ins are imple-

mented, speciﬁcally, the data acquisition components

to capture the data available for analysis and visual-

ization, but also the DTRON plug-in that bridges cog-

nitive functions to deliberativecontrol level functions.

The deliberativecontrol including overall safety mon-

itoring, interaction learning and action planning is

based on provably correct timed automata models.

3 DISTRIBUTED MODEL BASED

CONTROL

The SNR timed automata based action planning and

control make use of Uppaal tool suite(Behrmann

et al., 2004). That allows manual construction

of timed automata similar to visual programming

paradigm - using your computer mouse to click

around. Plus some limited functionality of C-like

functions to be used upon various elements of the

automata. Although the functions make it some-

what easy to specify state transitions their usage is

prone to state space explosion. The Uppaal tool-

suite includes an extension for Testing Real-time sys-

tems Online called TRON(Hessel et al., 2008)Al-

though TRON was originally developed for confor-

mance testing it also supports the functionality that

is relevant for discrete control To interface the TRON

model-based control stimuli with controllable object

requires “adapters” on the to be able to intermedi-

ate and interpret the signals trafﬁcking to and from

the automata. TRON is designed for single tester-

testee pair, but limits the scaling to n > 1 testers

and m > 1 testees. The limitation of TRON usage is

that it requires an extensive effort for adapter coding.

When the adapter-tester pairs are tightly coupled ev-

ery change in conﬁguration requires re-wiring on both

adapter ends .

Distributed TRON (DTRON) proposed in this pa-

per is a custom framework built around the TRON

tool to support multicast messaging between TRON

instances running in parallel. In the ISO OSI net-

working architecture sense it implements the white-

board pattern where the agents publish data and the

subscriber agents get notiﬁed about this. On the other

hand, it supports the dependency injection program-

ming paradigm to make the controller controllable

object pairs loosely coupled for better scaling. Mul-

ticast is a message sent not to one recipient but n re-

cipients. DTRON is able to intercept the designated

transitions within one control agent (practically in its

model) and inform the other control agents of inter-

ests about it. The designation is deﬁned by predicate

on a synchronized transition of the controlling agent

model.

This synchronization and communication be-

tween agents is implemented by means of multicast

message passing that allows the agents to join and

leave a multicast whenever they want without the

need to re-conﬁgure existing infrastructure. The con-

trol agents do not need to be re-programmed when

this happens. It only requires an agreement protocol

on how the messages are deﬁned and what data they

carry when they traverse the multicast.

4 CONTINUAL PLANNING AND

CONTROL

Continual planning(DesJardins et al., 1999) means

the planning strategy when all interactions are not

fully planned ahead, but reacting to the situation as

it emerges. The controller knows the state of the con-

trol object it tries to reach, but doesn’t have full con-

trol over the stimuli and behavior of it. Then the con-

MODEL BASED CONTINUAL PLANNING AND CONTROL FRAMEWORK FOR ASSISTIVE ROBOTS

405

troller stimulates the object step-by-step and reacts to

responses of the object to drive it towards the goal.

Timed automata based planning and control suits

for this kind of control scenario quite well due to its

non-deterministic nature. Observations can easily be

mapped to an automata locations and transitions en-

coding this non-determinism. Uppaal comes with a

formal veriﬁcation engine that can be used to establish

weather a “plan” always drives the object to a desired

state. An extreme case is a fully non-deterministic

automaton that implies that it cannot be guaranteed

or estimated which conditions should hold in order to

guarantee that a planned target state of the control ob-

ject is always reachable.

5 REACTIVE PLANNER

For planning and controlling the SNR action in non-

deterministic situations reactive planning controller is

synthesized on-the-ﬂy based on the interaction model

the SNR has learned by observing and recording

Scrub Nurse and Surgeon’s interactive behavior. The

timed automata model learning algorithm used for

that has been introduced in [vain2009humanrobot].

For synthesis of the reactive planning controller that

guides the SNR action when being active is based

on the interaction model the algorithm described in

Reactive testing of non-deterministic systems by test

purpose directed tester(Vain et al., 2011). Intended

control goal of the SNR operation is encoded in the

scenario automaton that speciﬁes the sub goals of the

control, their temporal order and timing constraints.

Whenever one of the sub goals has been reached it

triggers the reset of guard conditions in the interac-

tion model and activates driving conditions to reach

the subsequent goal or one of the alternatives if mul-

tiple equal goals are reachable. In case of violating

the timing constraints or blocking an exception han-

dling procedure or reset is activated and diagnostics

recorded. Special care has been taken to address the

safety precautions in SNR control. An independent

safety monitoring process is running to check if all

safety invariants are satisﬁed. Whenever safety viola-

tion is detected the emergency stop is activated.

6 CONCLUSIONS

The cognitive robot architecture framework described

in this paper supports several innovative aspects

needed for implementing assisting robots in differ-

ent applications. Our experience is based on the

Scrub Nurse Robot control architecture and software

platform development exercise. We demonstrate that

DTRON model-based distributed control framework

provides ﬂexible infrastructurefor interfacing data ac-

quisition and cognitive functions with the ones of de-

liberative control level planning and decision making.

The architecture merges also a module for learning

human interactions and model construction with re-

active planning controller generator and runtime ex-

ecution engine. The timed automata based interac-

tion model learning, on-the-ﬂy reactive planning con-

troller synthesis and online safety monitoring in SNR

are steps towards the concept of provably correct

robot design.

REFERENCES

Behrmann, G., David, A., and Larsen, K. G. (2004). A tuto-

rial on uppaal. In Bernardo, M. and Corradini, F., edi-

tors, Formal Methods for the Design of Real-Time Sys-

tems: 4th International School on Formal Methods for

the Design of Computer, Communication, and Soft-

ware Systems, SFM-RT 2004, LNCS, page 200–236.

Springer–Verlag.

DesJardins, M. E., Durfee, E. H., Ortiz Jr, C. L., and

Wolverton, M. J. (1999). A survey of research in dis-

tributed, continual planning. AI Magazine, 20(4):13.

Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann,

P., and Witten, I. H. (2009). The WEKA data min-

ing software: an update. SIGKDD Explor. Newsl.,

11(1):10–18.

Hessel, A., Larsen, K., Mikucionis, M., Nielsen, B., Petters-

son, P., and Skou, A. (2008). Testing Real-Time sys-

tems using UPPAAL. In Formal Methods and Testing,

page 77–117. Springer-Verlaag.

Miyawaki, F., Masamune, K., Suzuki, S., Yoshimitsu, K.,

and Vain, J. (2005). Scrub nurse robot system-

intraoperative motion analysis of a scrub nurse and

timed-automata-based model for surgery. Indus-

trial Electronics, IEEE Transactions on, 52(5):1227

– 1235.

Vain, J., Kull, A., K¨a¨aramees, M., Maili, M., and Raiend, K.

(2011). Reactive testing of nondeterministic systems

by test purpose directed tester. In Model-Based Test-

ing for Embedded Systems., Computational Analysis,

Synthesis, and Design of Dynamic Systems, pages

425–452. CRC Press - Taylor & Francis Group, Mas-

sachusetts, USA.

Vain, J., Miyawaki, F., Nomm, S., Totskaya, T., and Anier,

A. (2009). Human-robot interaction learning using

timed automata. In ICCAS-SICE, 2009, pages 2037

–2042.

PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems

406