Robotics Enabled Augmented Health

Artur Arsenio

IST-ID, Universidade da Beira Interior, YdreamsRobotics

Edifício A Moagem - Cidade do Engenho e das Artes, Largo da Estação 6230-311 Fundão

{artur.arsenio}@ydreamsrobotics.com

Abstract. Nowadays it is increasingly important, for social and economic rea-

sons, to provide augmented health assistance for people at home. This will

mostly benefit some specific user groups, such as elderly, patients recovering

from physical injury, or athletes. This chapter describes the application of ro-

botics, under the scope of the Augmented Human Assistance (AHA) project, for

assisting people health. Two complementary and interacting approaches are de-

scribed. The first consists on a mobile robot nurse, that assists patients and el-

derly on their daily lives, such as for advising on medicine intake, or providing

complementary biomedical sensing equipment for health monitoring. The other

approach consists of multimodal feedback assistance through augmented reali-

ty, haptic sensing and audio, in order to guide and assist people on the execu-

tion of exercises at home.

1 Introduction

The world population is aging rapidly. There is an increasing need for health assis-

tance personnel, such as nurses and physiotherapeutic experts, in developed countries.

On the other hand, there is a need to improve health care assistance to the population,

and especially to elderly people.

The World Health Organization estimates that sedentarism is the 4

main factor in

worldwide mortality, and is associated with several diseases, such as breast and colon

cancer, obesity, diabetes, and ischemic strokes. For instance, childhood obesity origi-

nates other health problems such as hypertension related to left ventricular hypertro-

phy, atherosclerosis and diastolic dysfunction. It is therefore important to identify

children in risk, through continuous health monitoring (e.g. body temperature, blood

pressure, electrocardiogram). Concerning elderly, estimates indicate that by 2030,

~4% of the USA population will have experienced a stroke, with related costs ex-

pected to rise from $71.55 billion to $183.13 billion between 2012 and 2030.

Hence, sedentarism is not only a social problem, but also economical, threatening

the sustainability of current health systems. New technologies are required for ena-

bling treatment at home, reducing pressure for health care personnel at hospitals.

Furthermore, automated systems at hospitals will enable significant cost reductions

and improved efficiency. Information technology has also been playing an increasing

role in the health care area throughout the years, aiding it to be more accurate, faster

to respond, and less susceptible to human errors. There is therefore the need not only

to develop technological solutions that promote active aging and prevent sedentary

behaviors, but also to find new technologies for assisting a growing, aging population.

Arsenio A.

Robotics Enabled Augmented Health.

DOI: 10.5220/0006162300330052

In European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems (EPS Lisbon 2015), pages 33-52

ISBN: 978-989-758-095-6

In this context, the advances in information, robotic and assistive technologies

have the potential to increase quality of life and change health care delivery models,

reducing costs, and improving monitorization. The “AHA: Augmented Human Assis-

tance” project is a novel, integrative and cross-disciplinar approach combining inno-

vation and fundamental research in the areas of human computer interaction, robotics,

serious games and physiological computing. AHA’s goal is to develop a new genera-

tion of ICT based solutions that have the potential to transform healthcare by optimiz-

ing resource allocation, reducing costs, improving diagnoses and enabling novel ther-

apies, thus increasing quality of life. The project proposes the development and de-

ployment of a novel Robotic Assistance Platform designed to support healthy life-

style, sustain active aging, and support those with motor deficits.

The AHA project will develop a novel and modular set of ICT based solutions that

in isolation or integrated as a Robotic Assistance Platform will address:

- Physical (re)training: Building on the existing expertise on Augmented Reality

(AR) and serious games, we propose to develop adaptive AR physical training

tools that deliver online feedback on performance to prevent sedentarism, sup-

port active aging and provide personalized tools for function re-training in motor

impaired patients.

- Increasing self-awareness: Monitoring of user state by means of biosensors,

computer vision systems and exercise performance data. User state will be as-

sessed in a transparent manner and data will be visualized through friendly user

interfaces, and shared with patients, clinicians and/or relatives.

- Augmented assistance: The above systems will be integrated on a mobile robotic

platform with indoor navigation capabilities (in environments such as senior

houses and hospitals) that will interact through a virtual coach system to assist

patients, provide reminders on tasks, guide patients through exercises, and sup-

port them in daily routines.

These technologies may be very useful in other scenarios, such as those of MOnarCH

project (Multi-Robot Cognitive Systems Operating in Hospitals). This CE FP7 project

is focused on introducing a fleet of social robots that will interact with sick children

and collaborate with medical staff on the pediatric ward of a hospital, the Portuguese

Oncology Institute of Lisbon (IPOL).

1.1 The Need for Robots

There are several scenarios in which robots and augmented reality are very useful in

the two aforementioned environments corresponding to these two abovementioned

projects.

• Teaching, for robots to supporting human teachers by projecting augmented

reality content on a wall or on objects.

• Patient rehabilitation exercises support, by projecting augmented reality con-

tent during physiotherapeutic activities in which the patient receives in real-

time visual feedback and corrective postures

• Information providers, such as projection of AR content informing people that

a new activity is about to start, or calling then with visual signs, or even mov-

ing along with people to places where action is going to take place (e.g. using

projected directional arrows), or informing someone to stop performing an

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

exercise.

• People protection, such as projecting augmented reality content (for instance a

stop sign) if a person moves into a forbidden zone or door, or performs a for-

bidden rehabilitation exercise or movement

• People entertainment: robots can play games with children or elderly people,

according to content projected into the floor. In another scenario, a patient re-

habilitation can involve game playing (serious games).

Furthermore, robots can perform several supporting activities at home, such as:

• support safe medicine delivery and intake. A robot may carry the medicine,

and at the appropriate time, take it to the patient.

• In addition, the robot may also carry on board medical equipment, such as

thermometer, arterial pressure measurement device, or electrocardiogram

(ECG) equipment, making it available to patients at home.

1.2 Remote Health Assistance

Ageing population has an enormous economic and social impact in a various areas,

especially healthcare systems. Elderly people are more vulnerable to physical or men-

tal impairments, disabilities and chronic illnesses. Falls and problems with muscle

bone can also limit the daily routines. Hence, such problems increase the potential

need for assistance of elderly people.

Patient health monitoring was traditionally done through periodical visits to the

doctor in order to undergo on-site tests for blood pressure, pulse, temperature, or

sugar level. The alternative is stationary monitoring, upon internship at a health care

provider. Currently, patients have the possibility to take home sensors attached on a

belt, usually for a specific time period, to collect biosignal data for that period. How-

ever, the patient still has to move to the health care provider to get the sensor, and

afterwards, he/she has to return to remove the sensor and deliver the data. Nowadays,

we are reaching seamless health monitoring by placing a mobile sensor on the patient,

executing biosignal collection, and transmitting data through a wireless access tech-

nology interface to a storage facility for further medical analysis [1].

On the other hand, medical assistance personnel play a very important role on pa-

tient recovery. In physical therapy, for instance, the therapist helps patients fighting

their pain and recovering from injury. His role is fundamental on therapy planning. In

addition, he not only demonstrates the correct execution procedure for the exercises,

but he also makes sure afterwards the exercises are executed correctly. With this goal

in mind, the therapist can intercede during the session and adapt the exercise schedule

according to the patient’s feedback [2]. However, the patient might perform incorrect

movements at home, without the therapist presence, to avoid pain [3]. Hence, there is

a need to provide assistive solutions that do monitor remotely the patients’ execution

of exercises, but also support, motivate and advise the patient to correctly perform the

exercises. The former is accomplished through biosensors, augmented reality and

haptic technology, that should monitor continuously the patient, transmitting wireless-

ly information concerning its health state and providing feedback. An interacting

mobile robot should provide the patient motivation and guidance.

Robotics Enabled Augmented Health

1.3 Chapter Structure

This chapter will start by presenting a review of previous relevant literature for aug-

mented health assistance. Afterwards, section 3 overviews the Augmented Human

Assistance project, namely its structure and challenges. Section 4 addresses the de-

sign of robot assistants targeting the AHA project user groups, which will act as nurs-

es for elderly people and patients in recovery of physical injury. It will be also shown

that robotics and augmented reality can provide further functionalities for providing

visual or haptic feedback to users, as described on section 5. Finally, section 6 will

draw the main conclusions, together with directions for future work.

2 Background Review

Various studies show evidence physically active elderly having lower rates of chronic

conditions such as cardiovascular diseases, diabetes, cancer, hypertension and obesity

[4]. There is the need, therefore, to develop solutions that promote healthy habits and

prevent sedentarism. Since chronic patients experience loss of autonomy and low self-

esteem, it is also important to provide assistance to patients with age related chronic

conditions. With these goals in mind, hereafter we overview previous research works

and discuss the most relevant strategies.

2.1 Service Robotics

Research interest in service robotics for active aging and health care has grown in the

last few decades with potential applications on healthy people, elderly, children or

patients. Robotic devices in elderly care [5][6], rehabilitation [7], autism diagnosis

and therapy [8] and weight loss applications [9] have been empirically demonstrated

to be effective. Hence, robotics raised great expectations on the use of robots as per-

sonal assistants. On such robot is the Nursebot platform, able to interact residents,

remind or accompany them of an appointment, as well as provide information of

interest to that person [10].

Intouch Health deployed their robot in a Neurology Intensive Care Unit. A study

suggested improvement in critical care nursing team satisfaction [11]. The robot Paro

in Japan (a robot resembling a baby seal, with expressive eyes) was reportedly able to

improve the mood of elderly people, and simultaneously reduced stress not only to

patients but also to their caregivers [12]. This has been demonstrated more recently to

treat some cases of depression suffered by the survivors of the devastating earthquake

and tsunami in the northeast coast of Japan in March of 2011.

Several solutions employing devices exist to support self-administration of medi-

cine and manage personal medicine administration [13]. The Kompai robot has been

tested for elderly assistance using a diary application for monitoring the medication

and give information about daily events [11]. The autom robot is a weight-loss social

robot [9] that asks questions concerning what a person ate, or how it exercised. It also

provides personalized, helpful suggestions and feedback using facial expressions and

a simple touchscreen interface. In Europe, the EMOTE (Robotic Tutors for Empathy

based Learning) research project explores the usage of social robots for teaching chil-

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

dren, using the NAO robot. This project has shown that the robot appearance, as well

as its motion, and functionality, plays a very important role on engaging the attention

of the learner [14]. Other authors incorporated the strict functional constraints im-

posed by complex environments (such as hospitals) and specific groups of users, into

the robot design process [15].

2.2 Biomedical Signal Analysis and Human State Estimation

Biomedical signal analysis is nowadays of greatest importance for data interpretation

in medicine and biology, providing vital information about the condition and affec-

tive/emotional states of subjects. In patients with neuromuscular diseases, a constant

monitoring of the patient’s condition is necessary [16]. Heart rate variability, respira-

tion, muscular and electrodermal activity signals are extremely important, since they

indicate when a muscular crisis is occurring.

Tracking devices are important to infer the human state, such as the posture and

motion of the human body during an activity (such as playing a sport or a therapeutic

exercise). Usually, several tracking points are used to represent human body joints

(with respect to both both position and velocity). Cameras have been used to detect

and estimate the pose of human subjects [17] and body parts [18], detect faces [19]

and their expressions [20] and, at a close range, detect eye movement and gaze direc-

tion [19]. Hidden Markov Models have been successfully used in gait recognition

[21].

One approach consists of using anthropometric proportions of human limbs and the

characteristics of gait, to achieve view-independent markerless gait analysis [22].

Microsoft’s Kinect is a markerless tracking device with an acceptable accuracy in

comparison with other motion tracking alternatives [23][24]. It provides full skeleton

tracking at low price. It is also easily portable, especially when compared to other

solutions requiring special equipment (e.g. markers) on the human body. A large

number of applications based on such sensors are addressing some of the difficulties

in unsupervised rehabilitation [25][26]. For instance, Gama et al. [27] proposed a

Kinect based rehabilitation system that tracks user position. The user sees himself on

the screen with overlaying targets representing the desired position. Real-time feed-

back using visual messages is provided in case of incorrect postures.

2.3 Rehabilitation Systems

Various rehabilitation systems have been proposed to improve patient recovery. Many

focus on specific injuries, e.g., stroke [25], or limbs rehabilitation [28][29]. Such

systems may have an important impact on patient’s rehabilitation on an ambulatory

scenario (e.g. at home). Enabling the patient to comfortably exercise at home im-

proves his motivation [25].

A patient’s rehabilitation is affected by exercise repetition, expert feedback, and

patient’s motivation [30]. The repetitive nature of rehabilitation exercises can quickly

become boring for a patient [28][31], therefore, there is a need of turning these exer-

cises into something less tedious. Indeed, successful patient recovery depends on

adherence to the scheduled planning [32].

Robotics Enabled Augmented Health

Repetitive exercises should be divided into several sub-goals, so the patient

achieves incremental success through each repetition. This improves motivation com-

pared to the approach where success is only achieved after finishing the whole task

[30]. Feedback can be given in two different ways, during the execution (concurrent

feedback) and at the end of exercise execution (terminal feedback) [2]. Concurrent

feedback is given in real-time for offering guidance or corrections in exercise execu-

tion. It allows the patient to have Knowledge of Performance (KP). Terminal feed-

back gives patients only Knowledge of Results (KR), since the patient receives feed-

back after fully executing the task [33][30]. Sigrist [2] suggests a temporal evolution

along the recovery phases. It proposes to gradually reduce KP, giving more emphasis

to KR, to stimulate patient’s autonomy.

2.4 Motor Training on Rehabilitation

The term exergaming is used for gaming approaches that motivate players to engage

in physical activity. Previous research [34] showed evidence that commercial tools

can produce physical, social, and cognitive benefits [35].

Unfortunately, these current tools are not suited to elderly or motor (re)training.

Motor rehabilitation, or motor re-learning, is an extensive and demanding process for

a patient, requiring discipline. Moving injured body parts may produce discomfort or

even significant pain [36]. Physical therapy sessions may be performed several times

both at a clinical and at home, or on either one of them. Later on, the patient might

have to continue therapy exercises at home [37] to avoid suffering a setback on reha-

bilitation [25] (or to decrease recovery time). This requires patients to learn the ap-

propriate recovery exercises. Furthermore, these exercises should be executed correct-

ly, to prevent an aggravation of the injury [3].

Repetition of specific movements is important for rehabilitation, whether at a clinic

or at home [30]. However, this is one of the main causes of deteriorated rehabilitation

at home, since patients tend to get bored and lose focus, due to the repetitive nature of

the task [36]. A mobile robot nurse should play a very important role here in order to

replace the therapist presence to guide and motivate the patient.

2.5 Augmented Reality

Augmented Reality (AR) is considered a promising technology in the rehabilitation

field. Several studies presented evidence for the benefits of employing Augmented

Reality (AR) techniques for supporting functional motor recovery [38], enabling ef-

fective motor training [39]. AR based approaches potentiate the combination of inter-

esting features such as training customization, extended multimodal feedback, quanti-

fiable therapy measures, extra motivation, among others [40].

Interactive AR applications have been proposed for different applications. Shader

Lamps [41] is a seminal work on AR. Its purpose was to augment an existing blank

model with computer-generated graphics to make the model exhibit realistic lighting,

shadow and even animation effects, thus providing it with characteristics previously

inexistent. Dynamic Shader Lamps [42] allows users to interactively change the gen-

erated graphics, and to digitally paint onto objects with a stylus.

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

There are various spatial augmented reality (SAR) projects with different applica-

tions. iLamp [43] supports SAR by casting the pre-distorted image of virtual objects

from the projector, which can capture the 3D environment based on structured pat-

terns. An interactive SAR system is described in [44] with multiple depth cameras

and a projector. The system detects the information of the surrounding environment

along with the users motion through depth cameras, and displays images on the sur-

face objects in a planar, real world table for the user to interact with. Pixelflex [45]

adopted an array of multiple projectors to make a huge screen space. Automatically

self-configured projectors are able to create a huge projection area.

A projection-based information display system was proposed in [46], which dis-

plays virtual images in a room with a series of sensors. The robot tracks the user

viewpoint with the sensors, and then the robot can generate anamorphic (i.e., projec-

tion format in which a distorted image is stretched by an anamorphic projection lens

to recreate the original aspect on the viewing screen), properly distorted images for

users on flat- surfaces. WARP [47] allows designers to preview materials and finished

products by projecting them onto rapid prototype models. The system uses a standard

graphical interface, with a keyboard and mouse used for user interface. The projection

is not restricted to a fixed area; all feedback is projected onto the tools. Surface Draw-

ing [48] allows a user to sculpt 3D shapes using their hands. Spray modeling [49] uses

a mock-up of a physical airbrush to allow the user to sculpt 3D models by spraying

matter into a base mesh. The system enhances physical tools by projecting status

information onto them allowing the overload of a single tool with several functions.

LightGuide [50] proposed projection mapping onto the user, using his body as a

projection screen. Different type of real-time visual cues are projected onto the user’s

hand to guide him for performing 3D movements. This way, the user is less suscepti-

ble to be distracted by external factors.

3 The Augmented Human Assistance Project

This section described the Augmented Human Assistance project. It is presented the

project structure of partners, its focus areas, as well as the main challenges and ex-

pected impact.

3.1 Project Partners

A consortium of key partners, addressing a large scope of technology fields, forms the

AHA project. IST-ID is the project coordinator. The Computer and Robot Vision Lab

(Vislab) research group at IST/Institute of Systems and Robotics addresses research

on Robotics, Computer Vision, and Cognitive Systems. The Carnegie Mellon Univer-

sity (CMU) team is part of the Quality of Life Technology Centre, at the Robotics

Institute, working on Artificial Intelligence and Human Computer Interaction. The M-

ITI NeuroRehabLab at University of Madeira addresses Serious Games, Interactive

Technology, and Rehabilitation. Two groups are specialized on biosignals data acqui-

sition and processing. The team at FCT/UNL works on Signal Processing, Machine

Learning, and Electrophysiology. And PLUX Biosignals, a company that creates

innovative solutions for Sports, Healthcare and Research, by integrating biosignals

Robotics Enabled Augmented Health

processing and miniaturized wireless sensor devices. YDreamsRobotics is another

company specialized in robotics, mechatronics and the internet of things, which is

addressing Augmented Reality, Robotics Design, and Haptic Feedback, for building a

therapeutic robot and for providing patient feedback through actuation. Finally, the

Interdisciplinary Centre for the Study of Human Performance (CIPER) at Faculdade

de Motricidade Humana (FMH) is specialized in the main areas addressed by the

project, namely Sports Science, Therapeutic exercises, and Human Function and Per-

formance.

The project targets three main environments (clinical, sports, and home assistance),

as described in the next subsection. Supporting external partners collaborate by

providing solution requirements, on-going feedback, as well as facilities for empirical

evaluation, as described in Table 1.

Table 1. External partners and their focus areas on the project.

Area External partners

Rehabilitation

HealthSouth of Sewickley

Clinical Physiology Translational Unit, IMM

Sports

Associação da Madeira de Desporto Para Todos

Via Activa Animação Turistica

Elderly

Comfort Keepers

Public Administration

Camara Municipal da Ponta do Sol

3.2 Project Focus Areas

The AHA project addresses three main target groups, as shown in Fig. 1. The Elderly

People group, addressed both on a clinical setting as well as at home. The goal is to

enable simultaneously sustainable care and active aging. New technologies are pro-

posed, such as medicine and medical equipment delivery by a mobile robot, and re-

mote assistance on therapeutic exercises using multimodal feedback and augmented

reality. Remote assistance on therapeutic exercises is also proposed for a second tar-

get group, those with motor deficits. Serious games, emotional and haptic feedback,

are among the technologies proposed to support patients on their recovery. Finally,

the project addresses sports for supporting a healthy lifestyle. It aims to improve ath-

letes’ performance, and to facilitate their learning, training and performance evalua-

tion using new technologies.

Fig. 1. User groups targeted by AHA: patients, athletes, and elderly people.

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

The three main target environments, namely clinical, sports facility, and home assis-

tance at a senior house, have different characteristics.

The clinical environment is more accessible but it is often a more complex envi-

ronment. A virtual coach module is proposed to interact with the patient, guiding him

throughout the exercises and providing positive and corrective feedback.

Sports facility is characterized by large indoor open spaces where robot navigation

is facilitated. The Robotic Assistance Platform will be able to identify an adequate

surface for the projection of the Augmented Reality Training Games, and provide its

users with the gaming training experience. Users will be able to challenge remote

users to a competitive game, while playing the training games.

At a senior house, the Robotic Assistance Platform will visit some of the residents

at specific times. The robot reminds the elder about the pills he needs to take, and also

asks him to take heart rate measurements with the on board sensors of the robot. Fur-

thermore, it will help the elder through his/ her gaming exercises, and it will provide a

feedback summary on performance, making this information also remotely available

to their clinicians.

3.3 Innovations and Impact

The AHA project proposes the development of novel robotics assistive platforms for

health exercise program management. Several functionalities will be provided, such

as user engagement, execution monitoring, and supporting therapists in patients’ re-

habilitation tasks. These functionalities will be developed considering the target user

groups, with the aim to promote a healthy lifestyle, a sustainable active aging of the

population, and supporting those with motor deficits. This should be accomplished

using several interdisciplinary technologies, and combining these in innovative ways.

For instance, it is proposed advanced “exergaming” and assistive technologies based

on human-computer interaction, robotics, serious games, multimodal sensing, physio-

logical computing, among other fields. The goal is to employ technology to help to

prevent sedentarism related diseases, to facilitate therapy of chronic conditions related

to aging and rehabilitation of motor deficits. Therefore, the project will tackle users

with special needs, focusing on elderly and patients with motor deficits, as well as on

athletes. It will provide customization and personalized tools for increasing motiva-

tion and engagement, using augmented reality, online feedback, gamification, social

networking, and interactive interfaces. It is also proposed a better measurement and

monitoring of user condition through multi-modal sensing (biosensors, computer

vision) and multi-modal feedback (haptic devices, sound feedback, augmented reali-

ty). Augmented assistance will be provided by a nurse mobile robotic platform, to-

gether with virtual coaching, for guiding patients in their exercises and daily routines.

The project also proposes to support clinicians through personalized user profiles and

advanced display of information.

The project outcomes aim to impact significantly different entities:

• Society: Reduce effects of sedentarism related diseases and aging conditions.

• Elderly: Increase physical fitness, independence, autonomy, self-esteem.

• Patients: Can exercise independently at home, adherence to schedule.

• Clinicians: Follow patient progress remotely, customize therapies.

Robotics Enabled Augmented Health

• Science & Technology: New algorithms and systems for eHealth technologies.

• Education: Formation of highly skilled human resources in close cooperation

with technology integrators and end-users.

• Companies: Close to market technologies; large exploitation opportunities.

3.4 WorkPlan

AHA project work plan is shown in Fig. 2. It represents the platforms, the modules to

be integrated on them, as well as the interactions among modules.

Fig. 2. Project workplan according to the different workpackages.

Human State Estimation (WP1) employs state of the art wearable technology and

training, for the integration of vision based activity monitoring and emotion detection

modules into a core technological and scientific component of the Robotic Assistant

Platform - the human monitoring module. Augmented Reality Training (WP2) inte-

grates AR technology that will serve as the basis for the development of a set of novel

serious games for physical training. Integration of Virtual Coaches (WP3) on a physi-

cal context is aimed for the improvement of the quality of life of those with special

needs. It includes (see fig. 3) user interaction aspects of the system, as well as interac-

tions of the virtual coach and the human monitoring modules. Robotic Assistance

Platform (WP4) consists of the sensing and software robot architecture. It includes

integration of the monitoring, AR training, and virtual coach modules into the final

Robotic Assistive Platform. User evaluation (WP5) will consist of the solution’s ex-

perimental validation with real data on a clinical context.

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

Fig. 3. Integration of interdisciplinary technologies on the AHA project.

4 Robot Design for Human Augmented Assistance

Recently, there is a trend for bringing robots into operations among people, helping

people. Elderly people, athletes and patients constitute challenging groups for robot

interaction. Addressing athletes’ needs is challenging due to their energy and high

performance requirements. Elderly people and patients pose strong safety concerns.

These challenges need to be tackled on the design process. This is essential for the

robots to be able to operate safely, while interacting socially with people.

Gonçalves and Arsenio [15] proposed a formal design process for building socially

interacting robots within a group of children with special needs in a hospital environ-

ment. The design takes into consideration the input of the project partners. This is

incorporated on the robot specifications during the development process. Robots de-

sign needs to account with several security factors. Under the scope of the AHA pro-

ject, we have been applying this process to the design of mobile robots targeting new

user groups. This “nurse” mobile robot, although posing similar challenges as the

ones described by Gonçalves and Arsenio [15], also adds new requirements. Elderly

people, athletes and patients poses specific requirements that often differ from those

for children in hospitals. Concerning security factors and human-robot interaction

features (see Fig. 4):

• Sick children often carry medical equipment carried, like wheeled structure to

carry serum bags. This still applies to some groups of elderly people. It may also

apply in less extent to patients on recovery from physical injury.

• No sharp surfaces that may cause injury to children. Although a less stringent

requirement for elderly people and patients, it is still an important issue.

• Avoid geometries that may invite children to step up the vehicle or insert fingers

on holes. This is not relevant for the group of users under consideration.

• Emotional expressions and engagement behaviors. A nurse robot targeting adults

should pose a professional stance, compared to a more character like concept for

children.

• Avoid inspiring fear and disgust: should instead induce comfort and trust, which

applies for all user groups (except in specific niches, such as military)

• An appealing aesthetics to children often has a different meaning than for an

adult.

Robotics Enabled Augmented Health

Fig. 4. Comparative analysis of security factors and human-robot interaction features, for ro-

bots to interact with children versus elderly people.

4.1 Shell Development Process

The shell development process is divided into several stages, and here briefly re-

sumed. It started by an analysis of the environment and the way elderly people, ath-

letes and patients should interact with the robots. At this stage we also performed

research related to existing robots, and “soft” and “clean” materials prone to be used

at clinical, hospital and sport facilities. With this information we concluded that

rounded shapes, soft materials, and neutral, light colors should be used, together with

active colors (red) on very small information symbols.

Different options were analyzed taking into account aesthetic properties, emotional

feelings provoked by the robot, its the actual expression, feasibility constraints and

functionality inherent to the robot, so that it could perform the envisioned tasks.

Afterwards, the material developed so far became the basis for the CAD model de-

velopment. The CAD model and photo-real imagery was also developed taking into

account the different production methods available, the assembly procedure, the

maintenance operations, the cost involved, and of course the aesthetics and functional

requirements defined at the early stages.

4.2 Operational Environment Analysis

The shell development process started by performing an analysis of the environments

where the robots will operate, including the deployment space, targeted groups of

people, and tasks to be executed by the robot, in order to define a set of features tak-

ing references on other existing and relevant systems.

The features to apply include rounded shapes and soft touch feeling materials, due

to the danger of fall by elderly people or patients recovering from physical injury.

This issue is also present in sport facilities due to high energy placed into activities.

The robots were given a dynamic stance, although this requirement has much more

relevance on sport facilities than on the senior houses or hospital/clinical environ-

ments.

The features to avoid include mechanical type shapes, which are more aggressive,

and not appealing for the target user groups. Exposed components, salient from the

robot body, are potentially dangerous and should be avoided as well. Robots outfit

 Consider medical equipment carried out by elderly

 Avoiding sharp features that may cause harm for elderly

 step up the vehicle or insert fingers on holes.

 special handles on the robots for guiding elderly people,

helping them moving around.

• Human-robot Interactions

 emotional expressions

 inspiring comfort and trust

 playful aspects.

• The aesthetics must now be appealing to elderly people,

taking references on a professional therapeutic or a butler.

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

should inspire users for interacting with them.

4.3 Ergonomics and Human Factors

Lets now consider the ergonomics and user factors, since problems can arise from bad

positioning and dimensioning of the robot components.

According to Fig. 5, we opted for: i) a non-threatening overall stance; ii) no arms;

iii) touch screen for better conveying user feedback, as well as to provide a simpler

user interface for elderly people.

Fig. 5. Mobile robot platform – the nurse.

Similarly to the Monarch robots (see Fig. 6), these should have a modern and cool

appearance. And since they operate in a hospital setting, there is a preference for

lighter colors (such as white color, common in hospitals), since it is easier to identify

dirty surfaces on robot and to clean them. Contrary to the Monarch robots, there is no

multifunctional face on the nurse robots.

Fig. 6. Mobile robot platform for children at hospitals, developed under the Monarch project

[15]. In contrast to the professional look and feel for the nurse robot, the monarch robots are

character-like, but not resembling any particular known artificial character.

5 Assistance Robotics for Augmented Health

Novel approaches have taken Augmented Reality (AR) beyond traditional body-worn

or hand-held displays, pushing such capabilities into mobile robots [51]. Spatial

Augmented Reality (SAR) uses digital projectors to render virtual objects onto 3D

Robotics Enabled Augmented Health

objects located in the robot’s navigation environment. When mounting digital projec-

tors on robots, this collaboration paves the way for unique Human-Robot Interactions

(HRI) that otherwise would not be possible. This is especially useful for supporting

patients performing recovery exercises, or for providing additional information to

elderly people (e.g. time for medicine intake).

5.1 Augmented Reality for Robotic Assistants

Fig. 7. Architecture of the proposed solution. The line in dash represents the connection to the

mobile robot localization system.

The architecture for SAR enabled robots is represented in Fig. 7. It is comprised of six

main categories:

• The hardware includes a projector, a camera, and a Kinect depth camera for hu-

man-robot interactions;

• The Kinect SDK interface, which allows direct access to Kinects RGB and depth

sensors. One of Kinects’ key features we used was skeleton tracking;

• Unity wrapper, which enables the use of Kinect SDK internal functions within

Unity;

• A Windows Presentation Foundation application (WPF) was developed in order

to simulate positional information retrieved from a robot localization system;

• Unity, the game engine on top of which the main applications were developed;

• The camera-projector calibration application, which makes use of ProCamCalib,

developed on Matlab.

A collection of modules (given by a series of scripts in Unity) were designed to per-

form specific tasks:

• Homography matrix update: updates the matrix associated with each projection

scene and apply that transformation in OpenGL’s vertex pipeline;

• Tracking: Controlling each projection surface position in the virtual scene;

• Intrinsic Setting: Alters Unity’s default projection matrix so that the values ob-

tained from the projection calibration process can be properly applied;

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

• Update Unity’s camera position: based on localization (position/pose)

information provided by an exterior application (on the robot or elsewhere);

• Save and load: of projection surface positional data from an XML file;

• Human-Robot Virtual Interface (HRVI): Updates the game logic of an

interactive AR game based on input received from Kinect’s skeleton tracking

information.

Fig. 8. Human interactions. One or two humans are tracked simultaneously using Kinect’s

skeleton tracking (step 1). The corresponding joint, for each player, controls the respective

virtual object (step 2). Camera 2 renders the texture containing the view of the camera (step 3)

that is then applied to one of the projection surfaces and updated in runtime (step 4). The final

step can now take place: project onto one real world surface (step 5).

Fig. 9. Augmented reality on Wheels – Content is projected onto multiple surfaces in order to

help user perform actions, or to play games with users (such as the Air Hockey game, as shown

in the right image).

Robotics Enabled Augmented Health

The goal is to project augmented reality content according to objects physics, or other

entities. As such, we introduce the human interactions feedback in order to adapt the

augmented reality content, and to achieve virtual-real objects integrated behavior, as

represented in Fig. 8.

Hence, the application integrates augmented reality information, with projection

distortion compensation, and human gestures recognition to enable interactions be-

tween people and undistorted augmented reality content. Virtual content is projected

onto the real world where humans interact with such content through movements (see

Fig. 9), as detected by Microsoft Kinect’s skeleton tracking feature.

5.2 Haptic Feedback

The AHA project is also investigating the usage of robotic elements, behind the nurse

mobile robot. Besides biosignal sensors on the human body (see Fig. 10a), actuators

are being investigated to provide real-time feedback concerning movements to be

executes, or corrections to imperfections on the execution of the exercises. Such actu-

ators include an elastic sleeve (see Fig. 10b), enabled with a sensor to measure joint

rotation and an array of motors to provide haptic feedback.

Currently two sensors have been tested: a Flex Sensor (see Fig. 11), a flexible strip

that senses rotation, as well inertial sensor based on gyroscope and accelerometer.

Other devices are planned to be tested, such as mobile phone gadgets (like Pebble),

that are able to generate controlled vibrations.

a) b)

Fig. 10. a)Biosignal sensing devices, to acquire data concerning biological variables on the human

body; b)Elastic sleeve prototype with a SensorFlex and two vibration motors for haptic feedback.

Fig. 11. Prototype schematics and testing, using a Flex sensor and two motors.

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

6 Conclusions and Discussion

This chapter presented some of the robotics technologies being developed for Aug-

mented Health Assistance, under the scope of the AHA project.

It has been shown that one of the problems with remote rehabilitation is the correct

execution of recovery exercises by the patient. Furthermore, due to the repetitive

nature of such movements, there is consequently a motivation problem for keeping

the patient engaged with the exercise program. The first issue can be addressed by the

usage of augmented reality and haptic feedback for providing corrective feedback to

the patient. The later issue can be addressed employing nurse robots. These replace

the role of the human therapeutic on providing advice and motivation. The nurse

robot also plays a very important role on supporting elderly people on their daily

activities. For instance, these robots can take medicine to elderly at intake times, or

make available medical equipment.

The AHA project proposed to develop new algorithms, and virtual as well as robot-

ic systems for eHealth technologies. It expects to have a significant impact on the

health and sport sectors, by employing interdisciplinary technologies at the service of

elderly, athletes and patients with physical injury (both at home or on a clini-

cal/hospital environment). Innovative approaches will bring benefits to society, by

reducing negative effects of sedentarism, such as related diseases and aging condi-

tions. It proposes as well to improve the quality of life of elderly people. Besides

developing a mobile robot acting as a companion nurse to elderly (reducing the prob-

lem of elderly isolation and loneliness), it also aims to increase elderly physical fit-

ness, independence from human nurses and family, and consequently improving their

autonomy and self-esteem.

Concerning patients in recovery of physical injuries, the AHA project proposes to

improve these patients quality of life. This is acomplished by enabling them to per-

form independently exercises at home, while receiving corrective feedback and moti-

vational incentive to adhere to scheduled exercises.

The project promotes however to keep clinicians in the loop (even remotely) to fol-

low patient progress. The clinician is also able to customize patients therapies, as well

as to provide of-line, or real-time, corrections to therapies or exercise execution.

The combination of a large set of interdisciplinary technologies is expected to ena-

ble a new level of health assistance to different user groups, improving their quality of

life and reducing government economic burden.

Acknowledgements. This work has been funded by CMU-Portuguese program

through Fundação para a Ciência e Tecnologia, project AHA-Augmented Human

Assistance, AHA, CMUP-ERI/HCI/0046/2013.

References

1. Arsenio, A., Andrade, J., Duarte, A.: Non-invasive Wireless Bio Sensing. Accepted for

publication at Lecture Notes in Computer Science, series Communications in Computer and

Information Science (CCIS), Springer-Verlag (2015)

2. Sigrist, R., Rauter, G., Riener, R., Wolf, P.: Augmented Visual, Auditory, Haptic, and multi-

modal feedback in motor learning: A review. Psychonomic bulletin & review 20, 1

Robotics Enabled Augmented Health

(Feb. 2013), 21–53

3. Tang, R., Alizadeh, H., Tang, A., Bateman, S., Jorge, J. Physio@Home: design explora-

tions to support movement guidance. CHI ’14 Extended Abstracts on Human Factors in

Computing Systems (2014)

4. World health statistics 2012. World Health Organization, 2012

5. Bemelmans, R., Gelderblom, G., Jonker, P., de Witte, L.: Socially assistive robots in elder-

ly care: A systematic review into effects and effectiveness. Journal of the American Medi-

cal Directors Association 13.2 (2012), 114-120

6. Broekens, J., Heerink, M., Rosendal, H.: Assistive social robots in elderly care: a review.

Gerontechnology 8.2 (2009), 94-103

7. Matarić M., Eriksson J., Feil-Seifer D.: Socially assistive robotics for post-stroke rehabilita-

tion. Journal of Neuro Engineering and Rehabilitation 4.5 (2007)

8. Scassellati, B.: How social robots will help us to diagnose, treat, and understand autism.

Robotics research. Springer Berlin Heidelberg, (2007), 552-563

9. Cory, K., Breazeal. C.: Robots at home: Understanding long-term human-robot interaction.

IEEE/RSJ IROS International Conference on Intelligent Robots and Systems, IEEE (2008)

10. Pineau, J., Montemerlo, M., Pollack, M., Roy, N., Thrun, S.: Towards robotic assistants in

nursing homes: Challenges and results. Special issue on Socially Interactive Robots, Robot-

ics and Autonomous Systems 42.3 (2003), 271-281

11. Rincon, F., Vibbert, M., Childs, V., Fry, R., Caliguri, D., Urtecho, J., Rosenwasser, R.,

Jallo, J.: Implementation of a model of robotic tele-presence (RTP) in the Neuro-ICU: ef-

fect on critical care nursing team satisfaction. Neurocritical care 17.1 (2012), 97-101

12. Wada, K., Shibata, T.: Robot Therapy for Elders Affected by Dementia. Engineering in

medicine and biology magazine, 27 (2008), 53–60

13. Wong, B., Norman, D.: Evaluation of a Novel Medication Aid, the Calendar Blister-pak,

and its effect on drug compliance in a geriatric outpatient clinic. Journal of the American

Geriatrics Society (1987)

14. Castellano, G., Paiva, A., Kappas, A., Aylett, R., Hastie, H., Barendregt, W., Nabais, F.,

Bull, S.: Towards Virtual and Robotic Empathic Tutors. 16

International Conference on

Artificial Intelligence in Education (AIED’13), Memphis, USA. In H.C. Lane & K. Yacef

(eds), Artificial Intelligence in Education Springer-Verlag, Berlin Heidelberg (July 2013)

15. Gonçalves, D., Arsenio, A.: Human-driven multi-robot design process for social interac-

tions with children on complex environments. 2015 6

International Conference on Auto-

mation, Robotics and Applications (Feb. 2015), 70–76

16. Pinto, A., Almeida, J., Pinto, S., Pereira, J., Oliveira, A., de Carvalho, M.: Home telemoni-

toring of non-invasive ventilation decreases healthcare utilisation in a prospective con-

trolled trial of patients with amyotrophic lateral sclerosis. Journal of Neurology, Neurosur-

gery & Psychiatry (2010)

17. Lim, J., Zitnick, C., Dollár, P.: Sketch Tokens: A Learned Mid-level Representation for

Contour and Object Detection. CVPR, Portland, Oregon (2013)

18. Girshick, R., Felzenszwalb, P., Mcallester, D.: Object detection with grammar models."

Advances in Neural Information Processing Systems (2011)

19. Arsenio, A.: Cognitive-developmental learning for a humanoid robot: a caregiver's gift.

MIT. PhD thesis (2004)

20. Yang, Y., Ge, S., Lee, T., Wang, C.: Facial expression recognition and tracking for intelli-

gent human-robot interaction. Intelligent Service Robotics 1.2 (2008), 143-157

21. Saponaro, G., Salvi, G., Bernardino, A.: Robot Anticipation of Human Intentions through

Continuous Gesture Recognition, CR-HRI (2013)

22. Goffredo, M., Seely, R., Carter, J., Nixon, M.: Markerless view independent gait analysis

with self-camera calibration, In Proceedings of International Conference on Automatic Face

and Gesture Recognition (2008)

23. Chang, C., Lange, B., Zhang, M.: Towards Pervasive Physical Rehabilitation using

Microsoft Kinect. 6

International Conference on Pervasive Computing Technologies for

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems

Healthcare (PervasiveHealth) (2012), 159–162

24. Scano, A., Caimmi, M., Malosio, M., Tosatti, L.: Using Kinect for upper-limb functional

evaluation in home rehabilitation: A comparison with a 3D stereoscopic passive marker

system. 5

IEEE RAS/EMBS International Conference on Biomedical Robotics and Bio-

mechatronics (Aug. 2014), 561–566

25. Borghese, N., Mainetti, R., Pirovano, M., Lanzi, P.: An intelligent game engine for the at-

home rehabilitation of stroke patients. IEEE 2

International Conference on Serious Games

and Applications for Health (SeGAH) (May 2013), 1–8

26. Kitsunezaki, N., Adachi, E., Masuda, T., Mizusawa, J.: KINECT applications for the physi-

cal rehabilitation. IEEE International Symposium on Medical Measurements and Applica-

tions (MeMeA) (May 2013), 294–299

27. Gama, A., Chaves, T., Figueiredo, L., Teichrieb, V.: Guidance and Movement Correction

Based on Therapeutics Movements for Motor Rehabilitation Support Systems. 14

Sympo-

sium on Virtual and Augmented Reality (May 2012), 191–200

28. Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie, J.,McDonough, S. Serious

Games for Upper Limb Rehabilitation Following Stroke. Conference in Games and Virtual

Worlds for Serious Applications (Mar. 2009), 103–110

29. Sadihov, D., Migge, B., Gassert, R.: Prototype of a VR upper-limb re-habilitation system

enhanced with motion-based tactile feedback. World Haptics Conference (WHC) (Apr.

2013), 449–454

30. Schonauer, C., Pintaric, T.: Full Body Interaction for Serious Games in Motor Rehabilitation.

Proceedings of the 2

Augmented Human International Conference, ACM Press (2011), 1–8

31. Rego, P., Moreira, P., Reis, L.: Serious games for rehabilitation: A survey and a classification

towards a taxonomy. 5

Iberian Conference on Information Systems and Technologies

(CISTI 2010).

32. Adherence to long-term therapies: evidence for action. World Health Organization (2003)

33. Huang, H., Ingalls, T., Olson, L., Ganley, K., Rikakis, T., He, J.: Interactive multimodal

biofeedback for task-oriented neural rehabilitation. 27

Annual International Conference of

the Engineering in Medicine and Biology Society, IEEE-EMBS (2005), 2547–2550

34. Garn, A., Baker, B., Beasley, E., Solmon, M.: What are the benefits of a commercial exer-

gaming platform for college students? Examining physical activity, enjoyment, and future

intentions. Journal of physical activity & health 9.2 (2012), 311-8

35. Staiano, A., Calvert, S.: Exergames for physical education courses: Physical, social, and

cognitive benefits. Child Development Perspectives 5.2 (2011), 93- 98

36. Singh, A., Klapper, A., Jia, J.: Motivating people with chronic pain to do physical activity:

opportunities for technology design. Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems (2014), 2803–2812

37. Lohse, K., Shirzad, N., Verster, A.: Video Games and Rehabilitation: Using Design Princi-

ples to Enhance Engagement in Physical Therapy. Journal of Neurologic Physical Therapy

37, 4 (Dec. 2013), 166–75

38. Laver, K., George, S., Thomas, S., Deutsch, J., Crotty, M.: Cochrane review: virtual reality

for stroke rehabilitation,” Eur J Phys Rehabil Med, 48.3 (Sep. 2012), 523–530

39. Cameirão, M., Bermúdez, S., Verschure, P.: Virtual reality based upper extremity rehabili-

tation following stroke: A review. Journal of CyberTherapy & Rehabilitation, 1.1 (2008)

40. Lucca, L.: Virtual reality and motor rehabilitation of the upper limb after stroke: a genera-

tion of progress?. Journal Rehabilitation Medicine, 41.12 (Nov. 2009), 1003–1100

41. Raskara, R., Ziegler, R., Willwacher, T.: Cartoon dioramas in motion. Proceedings of the

international symposium on non-photorealistic animation and rendering (2002), 7-13

42. Bandyopadhyay, D., Raskar, R., Fuchs, H.: Dynamic Shader Lamps: Painting on Movable

Objects. IEEE/ACM international symposium on mixed and augmented reality (2001)

43. Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., Forlines, C.: iLamps: Geo-

metrically Aware and Self-Configuring Projectors. ACM Transactions on Graphics (TOG),

ISSN: 0730-0301, 22.3 (July 2003), 809-818

Robotics Enabled Augmented Health

44. Wilson, A., Benko, H.: Combining Multiple Depth Cameras and Projectors for Interactions

On, Above, and Between Surfaces. In Proceedings of ACM UIST '10 (2010), 273-282

45. Yang, R., Gotz, D., Hensley, J., Towles, H., Brown, M.: Pixelflex: A reconfigurable multi-

projector display system. Proceedings of the conference on Visualization (2001), 167-174

46. Azuma, K., Miyashita, S.: Anamorphosis Projection by Ubiquitous Display in Intelligent

Space. Proc. int. conf. universal access in human-computer interaction (2009), 209–217

47. Harris, C.: Tracking with rigid models. In Active vision, MIT Press Cambridge, MA, USA

(1993), 59-73

48. Schkolne, S., Pruett, M., Schröder, P.: Surface drawing: creating organic 3D shapes with

the hand and tangible tools. Proceedings of SIGCHI Conference on Human Factors in

Computing Systems (2001), 261–268

49. Jung, H., Nam, T., Lee, H., Han, S.: Spray modeling: Augmented reality based 3D model-

ing interface for intuitive and evolutionary form development. Proceedings of the Interna-

tional Conference on Artificial Reality and Telexistence (2004)

50. Sodhi, R., Benko, H., Wilson, A.: LightGuide: projected visualizations for hand movement

guidance. Proceedings of the SIGCHI Conference on Human Factors in Computing Sys-

tems (2012)

51. Costa, N., Arsenio A.: Augmented Reality behind the wheel - Human Interactive

Assistance by Mobile Robots. 6

International Conference on Automation, Robotics and

Applications (Feb. 2015), 63 - 69

EPS Lisbon January 2015 2015 - European Project Space on Intelligent Systems, Pattern Recognition and Biomedical Systems