MODEL BASED CONTINUAL PLANNING AND CONTROL
FRAMEWORK FOR ASSISTIVE ROBOTS
A. Anier and J. Vain
Tallinn University of Technology, Ehitajate tee 5, 19086 Tallinn, Estonia
Keywords:
Assistive robotics, Model based control, Continual planning, Cognitive architecture.
Abstract:
The paper presents a model based robot planning and control framework for human assistive robots of medical
domain, namely for Scrub Nurse Robots. We focus on endoscopic surgery as one of the most relevant surgery
type for the use of robotic assistants. We demonstrate that our framework provides means for seamless inte-
gration of sensor data capture, cognitive functions, model based continual planning and direct actuator control.
The novel component of the architecture is a distributed continual planning system implemented based on the
Uppaal model checking and testing tool suite. The distributed and the contract-based modular architecture of
proposed framework enables flexible online reconfiguration and adaptability to various applications, but also
safe installation of new software components on-the-fly.
1 INTRODUCTION
The assistive robotics sets high standards to cogni-
tive capabilities, autonomy and movement precision
of robots. Intuitively, it means understanding hu-
man intention and adequate reaction to it. Techni-
cally, it means human-in-the-loopcollaborativeaction
control, fusion of various sensor information, high
accuracy actuation and reliable software implemen-
tation. Safety issues of action and trajectory plan-
ning become critical in the conditions where the robot
shares user’s working envelope and the contact due
to the expected physical interaction for transfer of
objects is frequent. This paper presents a new soft-
ware integration framework for a Scrub Nurse Robot
(SNR)(Miyawaki et al., 2005) focusing on distributed
model based continual planning and control issues.
The goal of a SNR is to learn the interactions be-
tween a surgeon and a scrub nurse during a laparo-
scopic surgery and to replace the (human) nurse on
demand. One of the important aspects of incorporat-
ing the SNR in the collaborative action, e.g. when the
human scrub nurse is occupied with other tasks avoid-
ing the need for the surgeon to re-adapt to the robot
partner and preserving the “original feel”. An exam-
ple of surgical scene with scrub nurse waiting for right
moment of handing over an instrument to surgeon is
depicted in Fig. 1.
A scrub nurse must hand a surgical instrument to
a surgeon as soon as it is requested. If the scrub nurse
Figure 1: SNR intraoperative scene.(Miyawaki et al., 2005)
has to spend time searching for the instrument after a
request, the procedure is interrupted, valuable time is
lost and an unnecessary burden is placed on the sur-
geon. That possibly reduces the quality and effective-
ness of the operation. To avoid such delays the scrub
nurse must be fully attentive to the activity in the op-
erative field and anticipate accurately what a surgeon
will need. For this to be possible the scrub nurse not
only needs to know the surgical procedure as well as
the surgeon does, but must also be highly disciplined.
The “ideal” scrub nurse (if one exists) is able to pass a
surgeon whatever is needed without any verbal order
at the moment that the surgeon’s hand is extended to
receive it.
The goal of SNR software project is to develop a
human-adaptiveSNR capable of adapting to surgeons
403
Anier A. and Vain J..
MODEL BASED CONTINUAL PLANNING AND CONTROL FRAMEWORK FOR ASSISTIVE ROBOTS.
DOI: 10.5220/0003827104030406
In Proceedings of the 2nd International Conference on Pervasive Embedded Computing and Communication Systems (PECCS-2012), pages 403-406
ISBN: 978-989-8565-00-6
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
with various levels of skill and experience, and even
to different personalities (and moods). In other words,
the SNR should be able to function as an “ideal” scrub
nurse. To attain this ideal, highly developed cognitive
faculties such as machine vision and speech recogni-
tion as well as adaptive robotic arm path planning and
targeting are required.
In conventional surgical operations a scrub nurse
frequently has to handle an array of different instru-
ments. It is very difficult to make the SNR adaptive to
such busy operations. Therefore, the SNR prototype
has been designed for endoscopic surgery which does
not need many types of surgical instruments. The
adaptivity of the SNR requires unsupervised learning
by observing skilled nurses’ actions and behavior dur-
ing surgical operations.
On-line recognition and anticipating surgeons
motions while operating is essential to classify which
motions are common to all surgeons and which are
specific to individuals. This, in turn, will aid in an-
ticipating a surgeon’s needs and in adapting to the
changes of procedure. On the other hand, the results
of the investigation of intraoperative behavior have to
be abstracted and memorized in the form of mathe-
matical and/or formal models in order to reproduce
the variety of motion trajectories that can be expected
from various combinations of surgical procedures and
varying external factors. The model of a nurse’s be-
haviors as he or she reacts to other surgical staff (sur-
geon, assistant and others) serves as a high-level be-
havior specification for the SNR action planning.
The SNR’s control architecture depicted in Fig.
2 comprises following components: MotionAnalysis
Hawk position tracking system that is capable of mea-
suring the position-tracking markers coordinates with
precision more than 1 mm with sampling rate up to
200 frames per second. The surgeon’s hand move-
ment sampling data is passed to gesture recognition
module that uses multiple recognition methods and
voting automata(Vain et al., 2009)for detection of sur-
geon’s current motion.The identified motion and its
parameters are inputs for reactive motion planning
that compares the observed movement of surgeon’s
hand with that of predicted by surgeon’s behavior
model and surgery scenario model, and makes the cor-
rection of the current model state, if necessary. By
the corrected state information and surgery scenario
model the next SNR action is planned and the result-
ing control parameters are transferred to the motion
control unit. The information about surgeon’s pos-
sible reactions predicted by the surgeon model is re-
turned to the motion recognition module to restrict the
decisions space when new movement is being recog-
nized.
Figure 2: Control loop outline.
2 SOFTWARE ARCHITECTURE
The control architecture described above is imple-
mented on the software framework written in C/C++
and Java and is designed around the Uppaal tool suite.
2.1 Data Acquisition
SNR doesn’t have integrated vision but makes use of
an external one. MotionAnalysis Hawk near-infrared
active 3D measurement system (3DMS) is used for
visual feedback. 3DMS is not the only source of in-
formation. There are various sensors to monitor the
state of the robot and peripheral interfaces that con-
tribute to the overall awareness of the environment.
Software architecture depicted in Fig. 3 unifies
3DMS data with other data acquisition sources for
easier integration with upper architecture tiers. This
includes capturing RFID information about the in-
strument positions in use. All the instruments are
equipped with ceramic RFID tags that allow easy de-
tection when the instrument is inserted to or removed
from trocar cannula. Video image from the laparo-
scopic camera is used for anticipation of surgeon’s
motions, but it is not discussed in this paper. Last
but not least, the middle-ware enables automated ex-
ecution of capture and training sessions for algorithm
evaluation, optimization and testing.
2.2 Data Analysis and Cognitive
Functions
The robot control framework provides a common
platform for integration of data acquisition and cog-
PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems
404
Computer mouse3DMS cameras
TRC
Socket
EvaComm2 SDK
Mouse Tracker
MotionAnalysis EVaRT
Visualization
Configuration
jEvart
Object definition subsystem
Rapid Miner
Uppaal TRON (DTRON)
Voting automata
TA automataTA generator
Spread Proxy
Filtering
Motion rec.
Spread op.
EVaRT op.
Filtering
Motion rec.
Spread op
EVaRT op.
Filtering
Motion rec.
Spread op
EVaRT op.
Spread
SNR
Figure 3: Architecture model.
nitive functions.
Data analysis and cognitive functions are imple-
mented by means of data mining toolkit Rapid Miner.
It includes hundreds of algorithms ranging from filter-
ing to machine learning packaged into an integrated
development environment. Rapid Miner is inspired
by WEKA machine learning toolkit(Hall et al., 2009)
but with extensive data visualization and analysis au-
tomation tools.
To make the Rapid Miner fit the SNR overall
control architecture some custom plug-ins are imple-
mented, specifically, the data acquisition components
to capture the data available for analysis and visual-
ization, but also the DTRON plug-in that bridges cog-
nitive functions to deliberativecontrol level functions.
The deliberativecontrol including overall safety mon-
itoring, interaction learning and action planning is
based on provably correct timed automata models.
3 DISTRIBUTED MODEL BASED
CONTROL
The SNR timed automata based action planning and
control make use of Uppaal tool suite(Behrmann
et al., 2004). That allows manual construction
of timed automata similar to visual programming
paradigm - using your computer mouse to click
around. Plus some limited functionality of C-like
functions to be used upon various elements of the
automata. Although the functions make it some-
what easy to specify state transitions their usage is
prone to state space explosion. The Uppaal tool-
suite includes an extension for Testing Real-time sys-
tems Online called TRON(Hessel et al., 2008)Al-
though TRON was originally developed for confor-
mance testing it also supports the functionality that
is relevant for discrete control To interface the TRON
model-based control stimuli with controllable object
requires adapters” on the to be able to intermedi-
ate and interpret the signals trafficking to and from
the automata. TRON is designed for single tester-
testee pair, but limits the scaling to n > 1 testers
and m > 1 testees. The limitation of TRON usage is
that it requires an extensive effort for adapter coding.
When the adapter-tester pairs are tightly coupled ev-
ery change in configuration requires re-wiring on both
adapter ends .
Distributed TRON (DTRON) proposed in this pa-
per is a custom framework built around the TRON
tool to support multicast messaging between TRON
instances running in parallel. In the ISO OSI net-
working architecture sense it implements the white-
board pattern where the agents publish data and the
subscriber agents get notified about this. On the other
hand, it supports the dependency injection program-
ming paradigm to make the controller controllable
object pairs loosely coupled for better scaling. Mul-
ticast is a message sent not to one recipient but n re-
cipients. DTRON is able to intercept the designated
transitions within one control agent (practically in its
model) and inform the other control agents of inter-
ests about it. The designation is defined by predicate
on a synchronized transition of the controlling agent
model.
This synchronization and communication be-
tween agents is implemented by means of multicast
message passing that allows the agents to join and
leave a multicast whenever they want without the
need to re-configure existing infrastructure. The con-
trol agents do not need to be re-programmed when
this happens. It only requires an agreement protocol
on how the messages are defined and what data they
carry when they traverse the multicast.
4 CONTINUAL PLANNING AND
CONTROL
Continual planning(DesJardins et al., 1999) means
the planning strategy when all interactions are not
fully planned ahead, but reacting to the situation as
it emerges. The controller knows the state of the con-
trol object it tries to reach, but doesn’t have full con-
trol over the stimuli and behavior of it. Then the con-
MODEL BASED CONTINUAL PLANNING AND CONTROL FRAMEWORK FOR ASSISTIVE ROBOTS
405
troller stimulates the object step-by-step and reacts to
responses of the object to drive it towards the goal.
Timed automata based planning and control suits
for this kind of control scenario quite well due to its
non-deterministic nature. Observations can easily be
mapped to an automata locations and transitions en-
coding this non-determinism. Uppaal comes with a
formal verification engine that can be used to establish
weather a “plan” always drives the object to a desired
state. An extreme case is a fully non-deterministic
automaton that implies that it cannot be guaranteed
or estimated which conditions should hold in order to
guarantee that a planned target state of the control ob-
ject is always reachable.
5 REACTIVE PLANNER
For planning and controlling the SNR action in non-
deterministic situations reactive planning controller is
synthesized on-the-fly based on the interaction model
the SNR has learned by observing and recording
Scrub Nurse and Surgeon’s interactive behavior. The
timed automata model learning algorithm used for
that has been introduced in [vain2009humanrobot].
For synthesis of the reactive planning controller that
guides the SNR action when being active is based
on the interaction model the algorithm described in
Reactive testing of non-deterministic systems by test
purpose directed tester(Vain et al., 2011). Intended
control goal of the SNR operation is encoded in the
scenario automaton that specifies the sub goals of the
control, their temporal order and timing constraints.
Whenever one of the sub goals has been reached it
triggers the reset of guard conditions in the interac-
tion model and activates driving conditions to reach
the subsequent goal or one of the alternatives if mul-
tiple equal goals are reachable. In case of violating
the timing constraints or blocking an exception han-
dling procedure or reset is activated and diagnostics
recorded. Special care has been taken to address the
safety precautions in SNR control. An independent
safety monitoring process is running to check if all
safety invariants are satisfied. Whenever safety viola-
tion is detected the emergency stop is activated.
6 CONCLUSIONS
The cognitive robot architecture framework described
in this paper supports several innovative aspects
needed for implementing assisting robots in differ-
ent applications. Our experience is based on the
Scrub Nurse Robot control architecture and software
platform development exercise. We demonstrate that
DTRON model-based distributed control framework
provides flexible infrastructurefor interfacing data ac-
quisition and cognitive functions with the ones of de-
liberative control level planning and decision making.
The architecture merges also a module for learning
human interactions and model construction with re-
active planning controller generator and runtime ex-
ecution engine. The timed automata based interac-
tion model learning, on-the-fly reactive planning con-
troller synthesis and online safety monitoring in SNR
are steps towards the concept of provably correct
robot design.
REFERENCES
Behrmann, G., David, A., and Larsen, K. G. (2004). A tuto-
rial on uppaal. In Bernardo, M. and Corradini, F., edi-
tors, Formal Methods for the Design of Real-Time Sys-
tems: 4th International School on Formal Methods for
the Design of Computer, Communication, and Soft-
ware Systems, SFM-RT 2004, LNCS, page 200–236.
Springer–Verlag.
DesJardins, M. E., Durfee, E. H., Ortiz Jr, C. L., and
Wolverton, M. J. (1999). A survey of research in dis-
tributed, continual planning. AI Magazine, 20(4):13.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann,
P., and Witten, I. H. (2009). The WEKA data min-
ing software: an update. SIGKDD Explor. Newsl.,
11(1):10–18.
Hessel, A., Larsen, K., Mikucionis, M., Nielsen, B., Petters-
son, P., and Skou, A. (2008). Testing Real-Time sys-
tems using UPPAAL. In Formal Methods and Testing,
page 77–117. Springer-Verlaag.
Miyawaki, F., Masamune, K., Suzuki, S., Yoshimitsu, K.,
and Vain, J. (2005). Scrub nurse robot system-
intraoperative motion analysis of a scrub nurse and
timed-automata-based model for surgery. Indus-
trial Electronics, IEEE Transactions on, 52(5):1227
– 1235.
Vain, J., Kull, A., K¨a¨aramees, M., Maili, M., and Raiend, K.
(2011). Reactive testing of nondeterministic systems
by test purpose directed tester. In Model-Based Test-
ing for Embedded Systems., Computational Analysis,
Synthesis, and Design of Dynamic Systems, pages
425–452. CRC Press - Taylor & Francis Group, Mas-
sachusetts, USA.
Vain, J., Miyawaki, F., Nomm, S., Totskaya, T., and Anier,
A. (2009). Human-robot interaction learning using
timed automata. In ICCAS-SICE, 2009, pages 2037
–2042.
PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems
406