Robot-Dog – Human Interaction in Urban Search

and Rescue Scenarios

Improving the Efficiency of Rescue Teams in Hazardous Environments

Anna Bosch

, Xavier Cufí

, Josep Ll. De la Rosa

and Albert Figueras

Institute of Informatics and Applications, University of Girona, Campus Montilivi EPS-4, 17071 Girona, Spain

Department of Computer Engineering, University of Girona, Campus Montilivi EPS-4, 17071 Girona, Spain

Keywords: Robot-Dog – Human Interaction, Autonomous Robot, Cognitive Systems, Improving Efficiency of Search

and Rescue Teams, Trained Dogs.

Abstract: After a natural urban disaster the interior of the rubble is often where the majority of victims are located.

Mortality rates increases and peaks after 48 hours, so it is of major importance to have fast and effective

search and rescue teams. Nowadays, the rescue and exploration teams normally use dogs as a companion to

find victims. Trained dogs are very helpful in these situations since their high mobility, speed and detection

capacity. However they need constant instructions and supervision, they can be in danger in some situations

and they are not able to collect precise data from the environment. Instead of trying to build competing

devices, the COMPANIONS project looks at cooperation between natural and artificial creatures and in

particular robots and dogs. This is rather new ground for research, where all the dog shortcomings can be

complemented with autonomous robots with cognitive abilities able to cooperate with dogs and humans in

search and rescue environments. The aim of the project is to analyse how a team of agents (robots-dogs-

humans in this case) can cooperate and interact during search and rescue. Research will be towards a new

rescue scenario composed that will allow: (i) to empower the best characteristics of all the involved agents

and to minimize the worst ones; (ii) provide the fundamental tools for enabling these three agents to work in

a cooperative and efficient way in rescue missions; and (iii) and to lengthen the human-dog link by allowing

the exploration combining mobile robots and trained dogs with more distant and safer human intervention in

the dangerous rescue scenes. The main challenge will be the dog-robot interaction: to give visual cognitive

and reasoning abilities to the robot in order to let him autonomously interact and cooperate with the dog

according its behaviour and the environment conditions; and to specifically train a dog to correctly accept

and interact the robot (in charge of an expert dog training company).

1 INTRODUCTION

After a tsunami, and earthquake, or any big World

urban disaster only a small fraction of victims may

actually survive. The majority of survivors (80%)

are surface victims and only 20% of them come

from interior of the rubble. The interior is often

where the majority of victims are located and

mortality rates increases and peaks after 48 hours,

meaning that survivors who are not extricated in this

first period of time are unlikely to survive beyond a

few weeks in the hospital.

Nowadays the rescue and exploration teams

normally use dogs as a companion mainly to find

hidden injured persons or victims. Trained dogs are

very helpful in these situations because of their high

mobility, speed and detection capacity. However

they need constant directions and supervision

provided by the rescue team members, they can be

in danger in some situations (e.g. if there are high

levels of undetected lethal gases) and they are not

able to collect precise data from the environment.

All these shortcomings can be complemented with

autonomous (or semi-autonomous) robots with

cognitive abilities able to work in dynamic, non-

deterministic rescue and exploration environments.

Robots have a high sensorial capacity, can collect

and interpret very precise data and can operate in

hazardous zones in an autonomous way.

We propose a new rescue approach composed by

a robot-dog-human team that will allow: (i) to

maximize the favourable characteristics of all the

366

Bosch A., Cuﬁ X., Ll. De la Rosa J. and Figueras A..

Robot-Dog – Human Interaction in Urban Search and Rescue Scenarios - Improving the Efﬁciency of Rescue Teams in Hazardous Environments.

DOI: 10.5220/0004117403660371

In Proceedings of the 9th International Conference on Informatics in Control, Automation and Robotics (ICINCO-2012), pages 366-371

ISBN: 978-989-8565-22-8

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

involved agents in the triangle formed by humans,

dogs and robots, and to minimize the adverse ones;

(ii) provide the basics of these three agents to work

in a cooperative and efficient way in rescue

missions; and (iii) and to enlarge the human-dog link

by allowing the exploration using mobile robots,

Canine Augmentative Technology (CAT) systems

and trained dogs limiting the human supervision.

Moreover there might be some daily situations

where not all the three agents are needed so the three

agents and the system developed should be able to

work independently.

The human-dog and the human-robot links have

been studied for years and are well documented in

the literature. However, and to our knowledge, it is

the first time that it is attempted to study the

interaction and cooperation between robot-dog

through an autonomous robot with cognitive

capabilities.

1.1 Improving the Performance of

Heterogeneous Search and Rescue

Teams

The goal of the project is to improve the

performance of trained Urban Search and Rescue

(USAR) canine teams. Note that the goal is to

improve the performance of the team as a whole.

This means that the performance of the humans must

be improved as the dog actually does very well on

their own. A dog moves through the rubble, detects

people and indicates where they are. Problems occur

when the handler cannot go where it goes and

therefore does not experience what the dog

experiences. This is where the cooperation with

rescue robots can be very helpful. The cooperation

and interaction of dogs-robots-humans in a USAR

scenario will have several advantages:

 A camera is normally used over the dog to

allow the human team to see what the dog is

looking at; however a camera over the robot

could bring new functionalities. It will allow

the human team (i) to see what the dog is

doing, (ii) to have more control over the whole

rescue scene, (iii) to interact with the dog if

he/her gets distracted or needs more

directions, and (iv) to interact with the victim.

 The visual information fused with the sensor

data interpretation will allow building a much

more precise situational awareness map.

 Mobility of dogs surpasses any robot mobility

(on rubble etc.) however a dog can work for a

very small period of time while a robot could

remain at the scene considerably longer.

 The space over the dog is limited and they

might be heavy sensors. A robot could bring

to the team multiple sensors that enhance the

search of the dog.

There are several works that showed that dogs

are not indifferent to robots (Lawson, 2005), so with

the proper training of dogs together with the right

motivation trough rewards, a successful interaction

and cooperation with robots is possible in the

opinion of the experts in Dog training (f.i. K9Dogs

Europe).

Concretely three agents are considered (i) a robot

called Link robot from now on; (ii) a trained dog;

and (iii) a human team. Brief descriptions of the

agents involved in the team and how they will

cooperate are as explained in section 2.

1.2 Specific Objectives

Dog Training and Interaction. Select and train a

dog for rescue situations. Train the dog for its

interaction with the robot(s). Study the best way for

a successful robot-dog communication. Build some

mock-ups for the Link robot so the training team can

analyse different solutions and configurations and

the dog can get used to it.

Agents Sensorization for the New Rescue

Scenario. Improve the limitations of the actual

Canine Augmentative Technology (CAT) systems

by studying a better distribution of sensors over the

dog. Investigate the best way to sensorize all agents,

dog and robot(s), and combine their sensors.

Give Visual Cognitive Capabilities to the Link

Robot. Localize and track the dog and recognize its

poses and actions. Dogs communicate through

innate responses and through learned signals. Much

can be learned about a dog’s state simply by

observing its body position and activity. Dogs have a

complex set of behaviours related to their social

position relative to other dogs and their physical, as

well as mental states. However, dogs can also learn

to communicate through barking and pose which

makes them ideal for USAR work as they can

roughly “tell” the handler what is going on when

they find something. Moreover the visual scene

interpretation could be used to build a situational

awareness map for the human team.

Build a Collaborative Map of the

Environment. Use different sensors (visual data, IR

information, sensor information) from the CAT

System, and Link robot to build a multi-layer map

useful to have complete information of the

environment from different sources. Give visual

Robot-Dog - Human Interaction in Urban Search and Rescue Scenarios - Improving the Efficiency of Rescue Teams in

Hazardous Environments

367

reasoning abilities for a better understanding of the

environment.

Give Autonomous Capabilities to the Link

Robot. Use ontologies, previous experiences and the

actual situation to give autonomy and decision

making to the Link robot. Design and develop

cooperative navigation and decision making

according to the information of all the agents (Link

robot, dog and humans).

Extend the Human-dog link by using the Link

Robot. Provide the communication tools and

interfaces so that the human team can control the

scene through the devices mounted over the robot(s)

and over the dog.

To Improve Performance, Safety and

Efficiency in Rescue Situations. The cooperation of

the three agents will improve on (i) performance,

each agent will use its best abilities in rescue

situations; (ii) safety, the robots will be equipped

with dedicated sensors to capture the rescue

environment and alert the dog and the humans in

case of danger; (iii) efficiency, the human team will

be able to supervise at the distance, having a global

idea of the situation so allowing them to act more

efficiently.

For the moment the objectives are limited to the

interaction of one dog with one Link robot and one

handler. The cognitive issues for handling crew of

several dogs, several Link robots, and several

handlers in complex rescue scenarios are out of the

scope of this initial proposal and are part of the

roadmap towards a full integration of crew

intelligence with dogs and robots presumably

affordable from 2020.

2 AGENTS OF THE NEW USAR

SCENARIO

2.1 The Link Robot

Vision cameras will be set on it to observe and

follow the dog and see what he/she is doing and to

give cognitive visual capabilities to the robot (dog

poses interpretation visual scene interpretation, etc.).

Moreover a laser will be used to point and show to

the dog where he/she has to look for (as humans do

nowadays) and it will carry a reward for the dog if

he/she finds a victim. It will be sensorized with a

screen, a camera and audio devices used to

communicate the human team with the victim and

with the dog, so used to enlarge the link with the

human team. It will carry much more sensors to

have a more complete map of the environment:

vision cameras, IR cameras, 360º cameras for visual

mapping; thermal and gases sensors will be put over

it so data analysis of them will be used to alert the

dog if a dangerous situation is detected (e.g. of

presence high level of lethal gases previously

undetected by humans). A visual map with the

merging and interpretation (high level reasoning) of

all the sensor information (vision and data) will be

build and send to the human team for their

awareness and supervision. This robot will work

autonomously in two ways: by following and

interacting with the dog and by exploring the scene

itself. Moreover humans can remotely control it if

necessary.

2.2 Trained Dog

A trained dog will be sensorized with Canine

Augmentation Technology (camera, audio and

wireless sent to the human team through the Link). It

will interact with the Link robot by: following orders

given by the human team, by searching where the

link robot is pointing, by obtaining a reward, by

making certain poses when a situation is done (e.g.

find a victim) so the Link robot can understand it. A

specific training of the Dog in order to be able the

interaction and cooperation with the robots will be

absolutely necessary.

2.3 Human Team

We will refer to the human team as the team on the

command area or control zone. They will have

access to the vision cameras (dog, link) to the audio

systems (dog, link), all this information will be sent

through the Link. It will have access to the map that

the Link robot will build using the link robot sensors

(visual, thermal and gases) and dog sensors (visual).

It will be able to see and speak with the victims

trough the Link robot (victims will be able to see

and speak with the human team). They will be able

to manually operate the Link and to speak and give

direction to the Dog through the Link.

Figure 1 shows the base of operations layout

defined by the International Search and Rescue

Advisory Group (INSARAG) with the new areas

and agents that will be introduced in our proposal.

INSARAG is a network of disaster-prone and

disaster-responding countries and organizations

dedicated to urban search and rescue (USAR) and

operational field coordination. Of special interest is

the Control zone that will be “improved” by giving

them more control on the rescue zone and a new

area to leave the robots.

ICINCO 2012 - 9th International Conference on Informatics in Control, Automation and Robotics

368

Figure 1: Base of Operations Layout defined in the

INSARAG Guidelines. Grey boxes indicate the new areas

and agents to be introduced.

3 TESTING SCENARIOS

To better understand the proposed research, we have

defined two scenarios, but more could be considered

during the development of this research:

Scenario 1 – Dog Searching. Dog and Link are in

the rescue zone, where the dog uses its abilities to

search victims and the Link follows him/her while

capturing data from the environment (visual and

sensors) interpreting the data and building a visual

map. This map and all the camera images of the dog

and the Link are sent to the control zone where are

analysed by the human team. The dog finds a victim

and acts with the pose of “victim found”, so the Link

interprets it using its cognitive visual capabilities

and rewards the dog. Link turns the screen on so the

victim can see and communicate with the human

team, and some first aid pack is given to the victim.

The human team can also talk to the dog to provide

it oral and visual positive feedback. The map is

updated with the location of the victim and some

data interpretation (e.g. some gases detection and

their degree of dangerous). The human team sends

further assistance to the victim.

Scenario 2 – Link Guiding. The Link points with

a laser where the Dog should go and search and they

both go there. The dog finds a void where he/she

cannot go in but the presence of a victim is detected,

so he acts with the pose “victim in the void”. The

Link recognizes the poses and alerts the human

team. The Link detects some gases that could be

dangerous, so updates the map that is sent to the

human control. Since gases could be dangerous for

the dog, Link robot alerts the dog and they both

leave for further exploration. The human team can

send assistance to the victim taking into account all

the information collected by the system (f.i. presence

of dangerous gases).

To achieve the generic intervention scenarios

presented above, it is necessary to advance in the

state of the art as it is explained in next section.

4 PROGRESS BEYOND THE

STATE OF THE ART

In the conventional search and rescue scenarios the

search team is composed by humans and trained

dogs. Recently, robots that are able to communicate

with humans are introduced in these scenarios,

however to the best of our knowledge there is still a

missing link: the dog-robot interaction, cooperation

and communication.

Within this research we propose to advance in

the state-of-the-art by providing cognitive abilities to

the robots so that they will be able to communicate

and cooperate with trained search and rescue dogs in

rescue context. Moreover the basis of the dog

behaviour for a correct communication with the dog

will have to be studied and researched. The new

proposed scenarios will have the three agents

mentioned above: dog, robot and humans where they

cooperate together. We will take into account all of

them in this research however special emphasis will

be given on the cognitive abilities for the Link robot

and the dog training for their cooperation.

This new workflow will require improvements

beyond the state-of-the-art on the fields of: dog

training, dog-robot interaction, dog sensorization,

dog localization and tracking, pose and action

recognition, scene understanding and object

recognition, data capture form the environment and

data fusion, high level reasoning and network

communications for sharing data between all the

agents. The advances in the technologies mentioned

will enable a new research workflow as proposed in

Figure 2

There have been other projects related to

different aspects of rescue robotics (GUARDIANS,

VIEWFINDER, NIFTI) but none of them considers

the animal-robot interaction, one of the key

objectives of this proposal. Regarding the Animal-

Robot Interaction we can find in the literature

several papers where a general interaction animal

robot has been attempted (Gribovskiy, 2010),

(Caprari, 2005), (Nanayakkara, 2008) and there are

several things to take into account for a correct

interaction between an animal (a dog in our case)

Robot-Dog - Human Interaction in Urban Search and Rescue Scenarios - Improving the Efficiency of Rescue Teams in

Hazardous Environments

369

Figure 2: Research areas; Dark grey boxes: strong

research; Soft grey boxes: integration; White boxes:

sensorization.

and a robot (Kim, 2009):

 The developer should have a full knowledge of

animal’s behaviours and emotions to make

robot understand animal’s needs well for natural

interaction with the animal.

 Robot should know the state of animal and

provide the service to animal by itself, as most

of animals do not know how to request a service

to robot.

 The animal may have fear of the robot because

the robot is a noisy creature, which has never

been seen before. The shape and texture of

robot need to be familiar with animal to reduce

the fear.

 The noise of the robot may astonish animal and

it would be a great difficulty for the robot to be

familiar with animal.

 As the sudden movement of robot may also

scare pet, robot behaviours should be well

designed considering the animal ethology.

Moreover in previous cited research works the

robots have very few (if any) cognitive ability that

gives them the ability to autonomously reason and

interact with the animal.

In respect to Dog Sensorization, the Ryerson

University team started to develop Canine

Augmentation Technology (CAT) (Ferworn, 2008),

(Ferworn, 2009). The CAT system currently consists

of side-mounted, armoured, low-light cameras with

on-dog recording and the potential of real-time

transmission of video, audio and telemetry

information. The notion of dropping items from

USAR dogs was devised by one of the Patent

holders of the Ryerson University Canine Remote

Deployment System (CRDS)-Kevin Barnum who

was a canine handler with the Ontario Provincial

Police at the time. The CRDS was modified with a

different kind of underdog designed around the idea

of a discarding "sabot" that holds a robot against a

dog's chest. This system is known as Canine

Automatic Robot Deployment System (CARDS)

(Tran, 2010). They have demonstrated various

robots being deployed including their own Drop and

Explore robot (DEX). In addition, CARDS relies on

the ability to detect and interpret canine barking in

order to drop a robot (the system detects and

interprets a dog’s barking). Of course CAT and

CARDS systems represent a very interesting starting

point of our research in this field. Moreover one of

the challenges we will face is that the useable space

on a dog is limited and variable. As such, some of

this research effort will be spent on reconfiguring

the CAT and CARDS systems to better distribute

sensors between the different involved agents

(robots).

It will absolutely necessary to recognize pose

and action recognition for the trained dog. We will

recognize static poses (such as “sitting down”) and

dynamic poses (such as “searching”). Assuming the

dog is well localized, recognized and tracked we will

first distinguish body parts and characterize those

using spatio-temporal features and template/shape

based features. A discriminative learning classifier

will be learned. Moreover since the robot can see the

pose from different point of view, 3D models will be

learned obtaining just 1 model for each pose or

action. 3D spatio-temporal models for pose and

action recognition will be studied. The scene context

will also be incorporated for a better action

recognition and understanding and to help

disambiguate similar poses and actions.

Finally a collaborative and enriched 3D map of

the environment will be built, benefiting from the

information provided by the different image

modalities and data sensors.

5 PROJECT STARTING POINT

The starting point of the idea presented in this

position paper is the work developed in the

framework of the Spanish national project “SAN

BERNARDO” funded by the “Ministerio de

Educación y Ciencia” that finished on 2010. The aim

of the project was to conduct research into control

and interactional architectures to carry out robot

cooperation tasks, whose purpose was to locate and

help people missed in emergency situations. In the

first step, these robots were used to locate survivors

and transmit their location to rescue services through

ICINCO 2012 - 9th International Conference on Informatics in Control, Automation and Robotics

370

wireless devices (to locate and communicate with

the rescue teams), and to render first aid kits to these

survivors. We proposed the research into awareness

in robots about their capabilities to perform tasks

entrusted to them, in order to ensure good

performance and cooperation functionalities in a

heterogeneous team of robots. In Figure 3 we show

this team of heterogeneous robots previously

developed for rescue scenarios. It includes: (i)

SmallBot. Its reduced size, small weight and high

speed (1m/s) do this robot easily transportable and

ideal for the first exploration after a catastrophe. (ii)

BigBot with a weight approx. 11kg that allows it to

evolve in more difficult lands and to avoid obstacles

up to 25cm. (iii) WaterBot is a robot thought to

operate in specific complicated lands with puddles.

Figure 3: Heterogeneous Robotic Team (Project SAN

BERNARDO). Spanish Research Agency.

REFERENCES

Gribovskiy, A., Halloy, A., Deneubourg, J.L, Bleuler, H.,

and Mondada, F., 2010 Towards Mixed Societies of

Chickens and Robots. IEEE/RSJ International

Conferenc on Intelligent Robots and Systems, October

12-22, 2010, Taipei, Taiwan, pp. 4722-4728.

Caprari, G., Colot, A., Siegwart, R., Halloy, J.,

Deneubourg. J.L., 2005. Animal and Robot Mixed

Societies, building cooperation between microrobots

and cockroaches. IEEE Robotics and Automation

Magazine, June 2005. Pp. 58-65

Nanayakkara, T., Dissanayake, T., Mahipala P., and

Sanjaya, KA.G., 2008. A Human-Animal-Robot

Cooperative System for Anti-Personal Mine

Detection. Humanitariang demining: Innovative

solutions and the challenges of technology.” February

2008, Vienna Austria.

Kim, J.H., Choi, S.H., Kim, D., Kim, J., and Cho, M.,

2009. Animal-Robot Interaction for Pet Caring.

Computational Intelligence in Robotics and

Automation (CIRA), 2009 IEEE International

Symposium on. December 15-18 pp. 159 – 164,

Daejeon, 2009.

Ferworn, A., 2008. Disaster Dogs.

Ferworn, A., 2009. Canine Augmentation Technology for

Urban Search and Rescue. Canine Ergonomics – The

Science of Working Dogs, William S. Helton (Ed.),

CRC Press Taylor and Francis Groupp, 2009, Boca

Raton, Florida, USA, ISBN: 978-1-4200-7991-3.

Tran, J., Ferworn, A., Gerdzhev, M., Ostrom, D., 2010.

Canine Assisted Robot Deployment for urban search

and rescue. IEEE International Workshop on Safety

Security and Rescue Robotics (SSRR), July 26-30,

Bremen 2010, pp. 1-6.

Lawson, Shaun 2005. Human.dog interaction as

inspiration for future assistance robots. International

Workshop on Designing robot applications for

everyday use. January 13-14

Robot-Dog - Human Interaction in Urban Search and Rescue Scenarios - Improving the Efficiency of Rescue Teams in

Hazardous Environments

371