Virtual Arm Representation and Multimodal Monitoring

for the Upper Limb Robot Assisted Teletherapy

Gorka Epelde

, Xabier Valencia

, Aitor Ardanza

, Elsa Fanchon

, Alessandro De Mauro

Francisco Molina Rueda

, Eduardo Carrasco

and Shabs Rajasekharan

eHealth & Biomedical Applications, Vicomtech-ik4, Mikeletegi Pasealekua 57, 20009 San Sebastián, Spain

EGOKITUZ: Laboratory of HCI for Special Needs, School of Informatics,

University of the Basque Country / Euskal Herriko Unibertsitatea,

Manuel Lardizabal 1, 20018 Donostia, San Sebastian, Spain

Biomechanics and Motor Control Laboratory LAMBECOM, Faculty of Health Sciences, Rey Juan Carlos University,

Calle Tulipán s/n, 28933 Móstoles, Madrid, Spain

Keywords: Armeo Spring, Exoskeleton, Rehabilitation Robots, Virtual Reality, Realistic Virtual Reality, Multimodal

Monitoring, Online Assessment, Upper Limb, Virtual Arm Representation.

Abstract: The use of technology in rehabilitation therapies targets the sustainability of health systems and the

improvement of quality of life of the user (therapists, patients and informal carers). Robot or exoskeleton

assisted rehabilitation systems, which are based on neurorehabilitation principles, are tools that not only

help patients move the arm with precision; they also help reduce the fatigue of the therapist during the

process. One of the challenges of the virtual reality based robot assisted upper limb rehabilitation is patients’

immersion within the therapy to achieve an improved progress of the rehabilitation.

This paper, presents a new virtual reality therapy that has been created using the Armeo Spring exoskeleton.

A 3D representation of the arm serves as an interaction mechanism with the virtual world. This makes the

user more aware of the movements that he/she is making and improves the rehabilitation outcomes. It also

encourages the user motivation and engagement to the therapy. Additionally, an application for the

multimodal monitoring of the patient has been developed, together with tools for the online assessment of

patients. These developments allow the physician to review the therapy without being in the same place and

time, optimizing the use of hospital’s human resources.

1 INTRODUCTION

Cerebrovascular accidents (CVA) and spinal cord

injuries (SCI) are currently the most common causes

of paralysis and paresis with reported prevalence of

12,000 cases per million and 800 cases per million,

respectively. Disabilities that follow CVA

(hemiplegia) or SCI (paraplegia, tetraplegia)

severely impair motor functions (e.g., standing,

walking, reaching and grasping) and prevent the

affected individuals from healthy-like, full and

autonomous participation in daily activities.

Moreover, the societal habits increase the

number of such episodes. The risk factors of the

today’s society are reported by the World Health

Organization at (McKay et al., 2004). Together with

the societal habits, the progressive aging of society

(World Heath Organization, 2011) calls to urgently

find new systems that will help mitigate the effects

of resource demand of such patients on the

healthcare system. In the path to guarantee the

successful implementation, deployment and use of

the solution, these systems should also improve the

quality of life of the involved people (therapists,

patients and informal carers).

One of the possible intervention points to palliate

the burden of healthcare systems is stroke

rehabilitation. Neurorehabilitation is a process that

aims to recover the capabilities to carry out regular

activities, lost by a neurological disease, by re-

learning or by active problem resolution.

Neurorehabilitation is based on the concept of

Neuroplasticity. According to experts, the brain is

able to establish new conections between neurons

Epelde G., Valencia X., Ardanza A., Fanchon E., De Mauro A., Molina Rueda F., Carrasco E. and Rajasekharan S..

Virtual Arm Representation and Multimodal Monitoring for the Upper Limb Robot Assisted Teletherapy.

DOI: 10.5220/0004642300690080

In Proceedings of the International Congress on Neurotechnology, Electronics and Informatics (VirtRehab-2013), pages 69-80

ISBN: 978-989-8565-80-8

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

that are able to subtitute lost sinapsis to a greater or

lesser extent. This is usually targeted by repetitive

therapies.

Traditionally, neurorehabilitation processes have

been based on the mobilisation of paralysed

extremities to avoid the stiffness and retractions of

joints and muscles. The most common therapy is the

Bobath method (Bobath, 1990) based on the

hierarchical organisation of the Central Nervous

System (CNS). According to Bobath, the direct

handling of the body at key points such as the trunk

is able to control afferent input and facilitate normal

postural reactions. According to this concept,

postures must be used to inhibit spasticity and tonic

neck reflexes and to facilitate righting and

equilibrium reactions.

With pass of time, new approaches have been

defined (Cano-de-la-Cuerda et al., 2012), such as the

task oriented rehabilitation process, which is based

on the functional task achievement, rather than on

the automatisms that are used for the task

completion. This method argues the repetition-based

therapies in the cases where they lack a functional

purpose. This approach seeks training cognitive,

psychological and sensitive aspects not taken into

account by the Bobath therapy.

The introduced therapies require several human

resources such as doctors and different therapists

(physical, occupational…). In most cases, therapists

need to be present during the therapy execution.

The latest advancements in robotics and

neuroscience have shown that robotic systems or

exoskeletons can facilitate functional task oriented

rehabilitation processes. According to several

authors, this type of therapy is more efficient for the

reduction of the effects of altered motor control.

Robot or exoskeleton assisted rehabilitation systems,

which are based on neurorehabilitation principles,

are tools that not only help patients move the arm

with precision; they also help reduce the fatigue of

the therapist during the rehabilitation process. This

type of rehabilitation is usually developed together

with a virtual reality environment. Additionally, this

type of system provides the therapists with tools to

make complete and objective studies over the

evolution of the patients. The key factors for a

successful robotic-assisted therapy have been

already described in a previous study (De Mauro et

al., 2012).

One of the challenges of the virtual reality based

robot assisted upper limb rehabilitation, is patients’

immersion within the therapy, to achieve an

improved progress of the rehabilitation. This

challenge is mainly motivated by the lack of a

realistic representation of the arm in the available

games. The lack of realistic representation is caused

by a lack of bioinspiration of the solution. This lack

of realism creates a lack of identification of the

patient with the virtual world, which negatively

affects the active role of the patient, and their

motivation. Therefore, the development of more

realistic arm models will improve the three key

factors (bioinspiration, active role of the patient, and

motivation) implied in the identified challenge.

Additionally, the analysis of state of the art on

robotic device based rehabilitation therapies has

underlined the need of remote therapy support and

the definition of a structured movement

quantification data format, interoperable with other

rehabilitation systems.

The objective of the present work is to develop a

realistic representation of the affected arm with the

aim to increase users’ motivation. Furthermore, the

research also targets the design and implementation

of the architecture and data structure to support

remote therapy and guarantee the interoperability

with other rehabilitation systems.

In section 2, a state-of-the-art in the field is

presented. In section 3, we detail our solution’s

design and implementation while in section 4 we

present the main results obtained from our solution’s

implementation. Conclusions and future work are

presented in section 5.

2 STATE OF THE ART

The lost of mobility or paralysis of upper extremities

is identified among the possible consequences of a

person suffering a stroke. The rehabilitation process

followed in these cases is composed of active or

passive training of the limbs and articulations, in all

the possible directions and with the largest run

possible. This process is usually done by performing

a large number of repetitions, which exhausts,

mentally and physically, both the patient and the

therapist. Therefore, the use of virtual reality (VR)

systems can help motivate the user. Currently most

of the available VR systems are not accessible to the

people with critical impairments, such as those who

have suffered from stroke or cerebral palsy. Thus,

the scientific community has worked in the use of

VR systems with the assistance of robotic systems or

exoskeletons.

In the following section, the different

technologies related to the identified research topics

will be studied. First, VR or games based systems

for rehabilitation, (including the design concepts)

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

will be analysed. Exosketons and robotic solutions

used for upper limb rehabilitation will then be

examined.

At the end of this section, technologies and

further considerations related to the remote

monitoring of the therapy support will be presented.

2.1 Virtual Reality based

Rehabilitation

The virtual reality systems can be classified as fully-

, semi- or non-immersive systems (Prashun et al.,

2010). The classification refers to the level of

perception of presence in a non-physical world and

the technology used to present the virtual world to

the user. A monitor, a projector or VR glasses are

normally used for each of these modalities.

Alternatives include the combined use of real

objects, augmented reality and VR to create a

mixed-reality (Hilton et al., 2011). This way, the

tasks of the rehabilitation solution can be brought

closer to real life tasks. This leads to a larger level of

cognitive learning and user acceptance, while

carrying out exercises to improve mobility. Hilton et

al., also report on studies that present the benefits

training of daily-living activities in VR scenarios

has, beside the benefits of the realisation of the

exercises, on applying the learned tasks in their real

life.

The main variables used for the rehabilitation

assessment are the time needed to get to the target

position, the speed at which the task is completed,

the accuracy of the trajectories and the range of

movements. The duration and the frecuency of the

therapy varies from study to study(from 45 minutes

to one or even two hours). Frecuency also varies

from 3-5 weeks upto 11-13 weeks with a repetition

of 5 days per week.

2.1.1 Rehabilitation Game’s Design

and Adaptation

For the design of rehabilitation games, many factors

need to be taken into account. One of the most

important ones is the feedback received by the user,

from the action being carrying out. This feedback

can be of different types: e.g. auditory, haptic or

visual. Feedback makes user motivation to increase,

either being to achieve a concrete task, achieve a

higher score or else (Maclean et al., 2000).

Different studies (Burke et al., 2010; Sveistrup,

2004) have underlined that the representation of the

user or his affected extremity in a virtual world is

very positive for accepting and engaging with the

therapy. In this type of system there is a continuous

feedback, so the user can focus on the affected

extremity, being conscious of the movements he/she

is doing.

Alankus et al. stress the importance of adapting

the games to each user and their evolution based on

different factors (Alankus et al., 2010). One of the

identified factors is that games should allow the user

a certain error margin and that errors consequences

should be constructive rather than catastrophic.

Another important factor is the game’s difficulty.

An excessive difficulty level can lead the user to

stop playing the games. On the contrary, an

excessive simplicity would get the user bored.

A study by N. Hocine et al. (Hocine and

Gouaich, 2011) shows that a dynamical difficulty

adaptation system, while maintaining a stable

success rate, influences positively the perceived

difficulty, making the therapy more satisfying and

rewarding for the patient. It is therefore important to

gradually increase the game’s difficulty, based on

patient’s evolution. Besides, the gradual increase of

difficulty attracts user’s attention. This distraction

can be used to facilitate patient’s pain management,

since when the patient is entertained, he/she is not so

conscious of pain.

Moreover, the games should also adapt to the

evolving user mobility capabilities. At the beginning

of the rehabilitation user’s mobility is very limited,

so the working area of the game should be reduced

compared to more advanced phases of the

rehabilitation process.

As rehabilitation involves a motor learning

process (Krakauer, 2006), the use of a taxonomy of

tasks, such as Gentile’s (Adams, 1999), can be

relevant here for both assessing the patient abilities

and building a coherent therapy plan gradually

increasing the movements complexity. According to

Gentile, four factors are to be taken into account: the

environment (whether it is changing in time and

space or not), the intertrial variability (if the target

changes between trials), the patient’s body motion

and the use of extremity. Other factors pointed by A.

Reinthal et al. (Reinthal et al., 2012) include whether

the task performed is continuous or discrete and

whether it requires unilateral or bimanual dexterity.

By increasing the movements’ complexity, the

patient gradually learns again how to perform

successfully activities of daily living, which involves

coordination between complex movements and

variations of the environment. In the case of

rehabilitation, intertrial variability is a key factor to

trigger a process of generalization of the

movement,for example for the patient to be able to

VirtualArmRepresentationandMultimodalMonitoringfortheUpperLimbRobotAssistedTeletherapy

grab an object regardless of its position (Krakauer,

2006).

Even if there is an increasing amount of games

targeted to rehabilitation, until now not all

entertainment possibilities present at mainstream

games are being exploited (Rego et al., 2010). For

example, there are no rehabilitation games, where

competitive, collaborative or net gaming is possible

among multiple users. These functionalities would

probably increase patients’ motivation.

2.2 Exoskeletons and Upper Limb

Rehabilitation Systems

The actual exoskeletons’ market is limited in

available models and their price is very high. These

are the main reasons for their low deployment.

Following, some of the commercially available or

scientifically reported exoskeletons are presented.

The Armeo Spring (Hocoma, 2013) is a

exoskeleton that embraces the whole arm, from

shoulder to the hand. This model is indicated for

those patients that have lost partially or completely

upper limb functionality, either caused by a brain,

spinal or muscular damage. This model has six

degrees of freedom (three in the shoulder, one in the

elbow and two in the wrist) without any robotic

actuator. Therefore, it is considered a passive

system. The Armeo Spring can be adjusted to patient

arm’s weight, so that patient doesn’t need to practise

any force to hold his arm. This way, a patient can

make use of his residual upper limb functionality.

With the aim of increasing the motivation and

acceptation of the therapy by the patient, the Armeo

Spring comes with different games that provide

auditory and visual feedback to the user. The

exercise area of the patient can be personalised to

the patient’s mobility range in each phase. Once the

game has finished, the session’s data is written into

an excel spreadsheet for patient’s evolution

assessment.

InMotion (InMotion, 2013) is an upper limb

(hand, arm, and shoulder) rehabilitation system. The

system is composed of three robotic systems:

InMotion Arm Robot, InMotion Hand Robot and the

InMotion Wrist Robot. These robotic systems have a

force-feedback system that helps users with mobility

limitations or users that lack coordination for

completing the objectives. As the patient evolves

and regains movement capabilities, the system

decreases the provided assistance in order to keep a

certain level of difficulty.

Qiu et al. defined a system (Qiu et al., 2010)

combining a force-controlled haptic interface

(Haptic Master) with a ring gimbal obtaining a six

degree of freedom, force-controlled robot targeted to

the upper limb rehabilitation. Adding the ring

gimbal as the end effector adds the possibility of

forearm rotation and records three more angles of

freedom. The system includes different games and

the possibility to limit the working area to adjust the

therapy to each patient’s capabilities.

With regard to exoskeleton systems based

combined therapies, August et al. defined a

rehabilitation system (August et al., 2011) that was

composed of computers, projectors, semi-transparent

mirrors, and light emitters to build a multisensory

therapy space, where the user can interact with both

real and virtual objects. The main idea behind this

type of therapy is that some users can lose the

perception of their body and their localisation while

participating in a virtual reality based therapy, which

is overcome with the integration of real object. The

mix of real and virtual objects can help improving

the motivation, as well as the auto-perception of the

body.

Some systems use the electromyography (EMG)

to monitor the biofeedback of the user. The objective

of this type of systems is to improve patient’s

abilities and his muscular activity. With this aim,

games are designed to use this type of biofeedback

to control some game elements. One of the games

(Ma et al., 2010) proposes to shoot balls to a target.

The proposed system for the game uses a movement

tracking system to move the shooter left-right and

up-down, while the EMG signal is used for shooting

the ball.

Similarly, other systems use brain computer

interaction (BCI). These systems are used to know

the emotional or the concentration state, to assist the

users with limb movement through robotic systems.

Ang et al. created a system (Angob et al., 2010)

combining a BCI system and a haptic robot (MIT-

Manus) for rehabilitation therapy. The users had to

make some initial tests for the system to identify the

signals related to their hand and arm movements.

After the initial signal training, neuro-feedback is

used to assist the user in completing the targeted

task. Even if the obtained results were worst than

those with the haptic robot itself, the help and the

information that these systems can provide to the

therapists, is clear. The information on the brain

activity complements the input for assessing

patient’s evolution.

2.3 Rehabilitation Remote Delivery

Telerehabilitation applications are categorized

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

(Brienza and McCue, 2013) into two main operating

models. The first application type is named as real-

time interactivity application, and it refers to those

applications that mimic face-to-face interactivity in a

medical facility. The second application type is

named as store-and-forward application, and follows

an asynchronous communication. A good insight

into the state of art of telerehabilitation systems can

be found at (Parmanto and Saptono, 2009).

Regarding the remote delivery of exoskeleton or

robotic system based rehabilitation, Oboe et al.

propose a haptic robot based remote rehabilitation

(Oboe et al., 2010). Their system allows the therapy

execution, even if the therapist and the patient are in

remote locations. The patient is provided with a

slave device, which transmits both the movement

and the applied force to the master device, hosted at

the therapist’s side. Since the haptic system works

two-ways, the therapist can assist or correct patient’s

movement.

Following, table 1 identifies the gaps and the

consequences detected during the state of art study.

Table 1: Gaps and consequences detected over the State of

the art.

Gap Consequences

Lack of realistic upper

limb representation in

the rehabilitation games.

Lack of patient’s

identification with the

virtual world. Lack of

motivation.

Lack of possibility to

carry out actions

remotely.

Need of specialised

personal in place.

The punctuations and

timing of the games are

not always significant

for the patient’s

evolution assessment.

Cannot objectively

evaluate patient’s

evolution.

Lack of exercise

recordings for offline

studies.

Systems do not allow

reproducing movement

or visualising patients,

neither locally nor

remotely.

Lack of standardised

movement data format

to compare it with other

tools.

Data cannot be

complemented with

information from other

systems, such as Kinect

based solutions.

3 SOLUTION’S DESIGN

AND IMPLEMENTATION

As presented in the previous section, traditional and

new robotic device based rehabilitation therapies

have some limitations. This paper’s implementation

tries to overcome the following:

 Lack of realistic upper limb representation

 Remote control / action impossibility and lack of

interactive online evaluation tool

 Lack of exercise recordings for offline studies,

lack of medical evaluation oriented measures, and

lack of standardised movement data.

3.1 Architecture Design

An architecture has been designed for coping with

the limitations identified. To guide the reader, the

defined architecture is depicted at Figure 1. The

designed architecture is composed of three elements:

Rehabilitation Centre (PC + Armeo), Service Layer

(PC) and Medical Professional’s Client (PC).

The Rehabilitation Centre is the responsible to

provide the user with the necessary games to carry

out the rehabilitation exercises. While the user is

doing the exercises, the system monitors and records

multimodal data. On the one hand, a webcam

records the user for an offline analysis of the

movement. On the other hand, joint’s movement

data is recorded for a more exhaustive study of the

exercises carried out by the user. Joint’s movement

data is later processed and presented graphically by

visualisation tools.

The Server layer stores and provides access to

the multimodal (data and videos) patient recordings

and the interactive online evaluation tool. This tool

has been developed using a visualisation toolkit and

Java Applet technology, with the aim to deploy it in

different platforms.

The Medical Professional’s Client is a graphical

rehabilitation sessions playback tool. From this

client, the medical professional is able to visualise

both the recorded data and videos. This client

integrates the server layer’s interactive online

evaluation tool, which graphically represents the

recorded data and allows evaluating data through

functionalities such as the distance measurement

between two points.

The developed architecture design, allows the

different tools to work independently from their

location. This enables robotic device based

rehabilitation therapists to conduct the therapy

remotely, not needing to have an expert present at

the rehabilitation sessions.

3.2 Rehabilitation Centre

The rehabilitation centre’s client is composed of an

VirtualArmRepresentationandMultimodalMonitoringfortheUpperLimbRobotAssistedTeletherapy

Figure 1: Architecture Design.

Armeo Spring exoskeleton, a PC, a webcam and the

developed software. The rehabilitation centre’s

software has been developed using C++

programming language, based on the Open Scene

Graph (OSG) library. The C++ programming

language has been selected to meet the Armeo

Exoskeleton application programming interface

requirements. OSG has been selected as the game

development platform, because it is open, it is a

high-level programming language and allows

loading models and animations with a base on

rotation transformation and allows the translation of

loaded objects. This geometric object manipulation

features facilitate representing a 3D arm, parting

from a robotic model of the Armeo. The

rehabilitation centre’s software is composed of three

modules: Games Module, Video Recording Module,

and Data Recording Module.

3.2.1 Games Module

The games module is the responsible for capturing

the movement data from the Armeo, for graphically

representing the user movements and for

implementing the games’ logic.

Movement Data Capture from the Armeo

Once the data is gathered from the Armeo, the data

is processed translating the gathered data into the

movements of the virtual arm. This process cannot

be publicly described, due to a confidentially

agreement signed with Hocoma, Armeo’s developer.

In case of needing related information, the reader

should contact Hocoma.

Graphical Representation

All the objects created for the virtual world have

been developed using the Autodesk Maya 3D tool.

Maya 3D has been selected due to the facilities it

provides to animate skeletons. Since most of the

developed objects are simple, the following

explanation is limited to the developed arm’s model.

For the movement generation of the virtual arm,

skeletal animation techniques have been used. These

techniques mainly consist on dividing the model in

two parts for animation development; the skin

representation and the hierarchical interconnection

of bones.

In order to develop a realistic arm model,

skinning technique has been used. This technique

consists on associating the bones with the vertices.

In some cases, the vertices can be associated to more

than one bone, therefore, weights are established so

the vertices act as real as possible. Figure 2 shows

the representation of the developed virtual arm.

Arm’s skeleton is represented in green, while the

vertices are indentified by the purple colour.

Figure 3 shows the weights influence. Weight

distribution in the wrist can be appreciated in the

form of a stronger white colour in areas where there

is more weight influence.

The virtual arms’ representation has six degrees

of freedom (DOF): two in the shoulder, two in the

elbow and two in the wrist, which allows

representing arm’s movements with certain fidelity.

To generate the movement of the virtual arm, the

data gathered from the Armeo has been processed to

obtain the angles at the six DOFs. From the

processed angles, rotations have been applied to the

respective points in the skeletons, through functions

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

Figure 2: Virtual Arm’s Representation.

implemented by the OSG library. Rotations, as well

as translations are automatically generated by the

OSG library.

The realistic representation of the arm permits

the user to be conscious of the movements he/she is

doing as shown by studies in immersive virtual

reality used with patients with phantom limb pain

(Murray et al., 2007). Many people who has suffered

a cerebrovascular accident, is not conscious of their

upper limb movements. Hence, a virtual

representation of their limb can help them seeing an

action and reaction effect. This effect can make

patients feel more identified with their limbs, which

can increase their motivation with the rehabilitation,

obtaining better results on the rehabilitation process.

3.2.1.3 Game’s Logic

With the aim of providing the user with a more

entertaining rehabilitation process, two games have

been developed. The games are based on the objects

reaching and grasping tasks. These tasks have been

selected because they are repeatedly used in users’

daily living activities.

In the first game, the user has to touch two balls

located on the left and right top ends. Once the user

touches the green ball, it changes its colour and the

other ball is changed to green, and so on and so

forth. Whenever the user completes a series of

touching both balls, a point is added to the marker.

A screenshot of this game is included in Figure 4.

The second game is based on a ball that changes

its location every time the ball is touched and adds a

point to the marker. Both games have a duration of

two minutes.

The main benefit of these games is having the

user conscious of his movements, thanks to the

virtual representation of his arm. As the patient

evolves and it is capable of doing movements easier,

the mobility range can be increased.

Figure 3: Weight’s distribution in the wrist.

3.2.2 Video Recording Module

The Video Recording Module records video of the

rehabilitation sessions, for its later offline study. The

development of this module has been based on the

Open Source Computer Vision Library (OpenCV

Library). The OpenCV library is a multiplatform

Figure 4: Rehabilitation Game.

VirtualArmRepresentationandMultimodalMonitoringfortheUpperLimbRobotAssistedTeletherapy

library targeted to real time image processing.

Additionally, OpenCV has supplementary libraries

for machine learning, that implement different

algorithms, such as k-nn and Naive Bayes.

The Video Recording Module is started

whenever the user selects one of the available games

and is stopped when the user finishes playing the

game. The video is recorded in a separate thread of

the application to avoid interference with the games.

Videos are recorded in colour at a resolution of

640x480 and 25 frames per second, which can be

easily modified.

The video recording is done through a webcam

connected to the PC controlling the Armeo

exoskeleton and placed in front of the Armeo.

3.2.3 Data Recording Module

Patient’s movement data is recorded by the Data

Recording Module. As for the video recording, the

movement data is recorded while the user is playing

a game. The data is saved in XML format. For the

XML structure handling, the Xerces library has been

used.

The Xerces library allows to do many XML

related tasks (e.g. parse, validate and handle). The

library implements the main parsing standards.

Since the main targeted use of the data collection

is the visualisation and evaluation of graphical data,

the selected XML format for structuring the

movement data has been GraphML. GraphML

enables to specify data following a graph structure

description approach. Additionally, is possible to

add specific information required by the application.

The resulting data structure is depicted on the left

side of Figure 5. A person can have data from

exercises done on different days. The exercises

contain data on the movement and position of

interest. The regions of interests are the shoulder, the

elbow and the hand. Each joint has an associated

graph, which is composed of measurement nodes

with values on the X, Y and Z axis.

On the right side of Figure 5, an example of the

defined interoperability structure is given. This

example shows the file’s namespace definition, user

data, the exercises carried out, and the results of the

shoulder’s movement data recording.

3.3 Service Layer

3.3.1 Interactive Online Evaluation Tool

With the aim of deploying a multiplatform

interactive tool for patient evaluation, an interactive

online evaluation has been developed using a

visualisation toolkit and Java Applet technology.

After an analysis of different visualisation

toolkits, Java Prefuse library was selected, due to the

capability of running developments as part of Java

Applets. The developed tool is able to compare the

points obtained for the joints, during the different

sessions of the therapy.

Each session of each game is represented in a

different colour and it is possible to choose both the

Figure 5: Data interoperability XML structure.

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

days and the axis for visualisation. Moreover, it is

possible to zoom and move through the display to

analyse a specific region in detail. A button of the

application allows going back to initial visualisation,

centring the image to the workspace of the Applet.

In addition to the presented functionalities, the

Applet is capable of calculating the distance between

two nodes by selecting them in the display.

The Applet works by reading the following

parameters: file’s directory, username, timestamp,

game, joint and axis. Based on these parameters, the

XML is read and organised in classes for easier data

manipulation and the Applet executes the actions

needed for the required graph visualisation.

Screenshots of the tool are given in the Medical

Professional’s client section where it is integrated in

a final application.

3.3.2 Web Services

The service layer has been defined to be as

compatible with different platforms as possible.

Therefore, for the development of this layer, Web

services technologies have been used and deployed

over an Apache Tomcat server.

Three services have been defined: (1) To store

patient’s session’s XML in the server; (2) To modify

the script parameters that launch the Applet; (3) To

obtain the urls for the recorded videos.

The client enters as parameters the

username/password and the XML file with the data

recording. The server is responsible for storing and

organising the data in the server.

The Medical Professional’s Client integrates

with the web service to modify the script parameters

that launch the Applet. This integration allows the

medical professional to choose easily the graph

he/she needs to visualize. The client sends the

script’s parameter he/she wants to modify and its

corresponding value as an input to the Web Service.

In addition, the Medical Professional’s Client

needs to load the videos paths to play them in the

display. For this task, username and the time range is

sent to the Web Service and the response contains an

array with the paths to the recorded videos.

3.4 Medical Professional’s Client

The requirements defined for the Medical

Professional’s Client were that the client should be

able to play rehabilitation session videos and the

visualisation Applet without the user having the

feeling of using an external application.

After testing different technological options, the

application was developed using C# programming

language, with .NET Framework version 2.5.

This technology has allowed creating the

Medical Professional’s Client, playing videos and

visualising data natively. From the user interface, a

target patient name is selected. The application then

communicates with the server and obtains the videos

and the visualisation data with the default

parameters.

Following, Figure 6 presents two screenshots of

the Medical Professional’s Client. On the image on.

Figure 6: Medical Professional’s Client: Analysis Data tool (left) and recorded video in the execution of the rehabilitation

tasks (right).

VirtualArmRepresentationandMultimodalMonitoringfortheUpperLimbRobotAssistedTeletherapy

the left, the data visualisation functionality is shown

on the image on the right the functionality of

rehabilitation sessions’ video playback is shown.

4 RESULTS

The main result of the presented work is an

exoskeleton based telerehabilitation platform, which

targets the improvement of user’s motivation and

adherence to therapy. The developed platform is a

preliminary answer to the gaps detected in the study

of the state of the art.

The representation of the patients arm in a virtual

world achieves a larger identification of the patient

with the virtual world and a larger level of

consciousness of his movements, with the benefits

of motivation and progress it implies. The developed

games, allow carrying the repetitive task in a more

entertaining way.

Through the new tools the doctors will be able to

be either present at therapy execution or guide the

therapy remotely. This allows delegating the therapy

setup to less qualified personnel if needed.

The provision of online tools allows improving

the care processes at hospitals which helps the

sustainability of healthcare systems. Furthermore, it

allows experts from different hospitals to analyse the

results, building better diagnostics and therapies.

Once the user has completed a therapy session,

the specialised personnel can remotely evaluate both

the recorded video and the visualisation of

movements. Each joint and exercise has its graphical

representation and is possible to select the days to be

shown and to compare the evolution of the selected

joint during the selected day range.

The data recorded by the platform is available to

other programs, which enables the comparison of the

results with other rehabilitation platforms. This

allows validating the obtained information with

other experimental systems such as the Kinect based

rehabilitation systems. Additionally, the definition of

an interoperable data structure enables the

combination of different rehabilitation therapies,

allowing different device (Armeo vs Kinect) based

rehabilitation therapies for hospital and home.

5 CONCLUSIONS AND FUTURE

WORK

The characteristics of the developed rehabilitation

platform can help patients, doctors and the

healthcare system, as presented in the results section.

But, in order to validate the real impact of the

development, the system needs to be tested with real

patients.

As introduced in the previous section, it is

important to integrate the developments with other

rehabilitation systems such as the ones based on the

Kinect for the development of an integral approach

to the rehabilitation. This way, the user would be

able to continue the rehabilitation therapy at home,

as an extension to the therapy started at medical

facilities.

The cost of acquiring an Armeo Spring or similar

robotic systems is not a cost that all hospitals are

able to assume. Therefore, research is needed on

tools for hospitals, whose cost is not too elevated. In

addition, a detailed analysis of the economic and

quality of life of the involved persons, comparing

the robotic system tools with conventional therapies,

can help clarify the real benefits of these systems,

for all involved stakeholders.

Regarding the improvement of the platform, new

functionalities to ease the medical professional’s

work could be implemented. For example, hiding

points not contributing to evaluation, adding a

functionality to delimit the areas for calculating the

mobility range, an automatic comparison between

sessions, etc.

With regard to the video visualisation, an

implementation of video on demand and streaming

technologies will improve the content management

and the performance of the platform. Additionally,

image analysis algorithms implementation could

help complementing the movement data captured.

Another interesting research line identified

during the present research is the development of

collaborative and competitive game modes for

robotic system based rehabilitation. These games

could be deployed over the net or at medical

facilities where robotic system based rehabilitation

is being carried out. Nowadays, there are many

successful conventional online games, which lead

the authors to think that the life collaboration or

competition with other patients can increase their

motivation and as a consequence improve their

progress.

Due to the fact that each stroke patient has

different needs and preferences, games should be

customisable, so the virtual world or its containing

objects could be adapted. This way, games could be

adapted to the specific preferences and needs of each

person, which would improve the resulting

rehabilitation process.

Last but not least, due to the limited scope of the

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics

presented research, the work on security

implementation has been limited. Taking into

account the confidentially and security needs of

using health data, the security aspects of this

implementation should be revised for real scenario

deployment.

ACKNOWLEDGEMENTS

This work was partially funded by the Spanish

Government CONSOLIDER-INGENIO 2010

Programme (Hyper Project).

REFERENCES

Adams, D. L. (1999). Develop Better Motor Skill

Progressions with Gentile’s Taxonomy of Tasks.

Journal of Physical Education, Recreation & Dance,

70(8), 35–38. doi:10.1080/07303084.1999.10605704.

Alankus, G., Lazar, A., May, M., & Kelleher, C. (2010).

Towards customizable games for stroke rehabilitation.

In Proceedings of the SIGCHI Conference on Human

Factors in Computing Systems (pp. 2113–2122). New

York, NY, USA: ACM. doi:10.1145/

1753326.1753649.

Angob, K. K., Guan, C., Sui Geok Chua, K., Ang, B.-T.,

Kuah, C., Wang, C., … Zhang, H. (2010). Clinical

study of neurorehabilitation in stroke using EEG-

based motor imagery brain-computer interface with

robotic feedback. In Engineering in Medicine and

Biology Society (EMBC), 2010 Annual International

Conference of the IEEE (pp. 5549–5552).

doi:10.1109/IEMBS.2010.5626782.

August, K. G., Guidali, M., Sellathurai, M., Jenu, S.,

Bleichenbacher, D., Klamroth-Marganska, V., …

Riener, R. (2011). A system for sensory motor

rehabilitation of the upper limb with virtual reality,

exoskeleton robot, and real objects. In Technologies

for Practical Robot Applications (TePRA), 2011 IEEE

Conference on (pp. 54–63). doi:10.1109/

TEPRA.2011.5753482.

Bobath, B. (1990). Adult hemiplegia: evaluation and

treatment. Butterworth-Heinemann Oxford.

Brienza, D. M., & McCue, M. (2013). Introduction to

Telerehabilitation. In S. Kumar & E. R. Cohn (Eds.),

Telerehabilitation (pp. 1–11). Springer London.

Retrieved from http://dx.doi.org/10.1007/978-1-4471-

4198-3_1.

Burke, J. W., McNeill, M. D. J., Charles, D. K., Morrow,

P. J., Crosbie, J. H., & McDonough, S. M. (2010).

Augmented Reality Games for Upper-Limb Stroke

Rehabilitation. In Games and Virtual Worlds for

Serious Applications (VS-GAMES), 2010 Second

International Conference on (pp. 75–78).

doi:10.1109/VS-GAMES.2010.21.

Cano-de-la-Cuerda, R., Molero-Sánchez, A., Carratalá-

Tejada, M., Alguacil-Diego, I. M., Molina-Rueda, F.,

Miangolarra-Page, J. C., & Torricelli, D. (2012).

Teorías y modelos de control y aprendizaje motor.

Aplicaciones clínicas en neurorrehabilitación.

Neurología, (0), -. doi:10.1016/j.nrl.2011.12.010.

De Mauro, A., Carrasco, E., Oyarzun, D., Ardanza, A.,

Frizera-Neto, A., Torricelli, D., … Florez, J. (2012).

Advanced Hybrid Technology for Neurorehabilitation:

The HYPER Project. In T. Gulrez & A. Hassanien

(Eds.), Advances in Robotics and Virtual Reality SE -

4 (Vol. 26, pp. 89–108). Springer Berlin Heidelberg.

doi:10.1007/978-3-642-23363-0_4.

Hilton, D., Cobb, S., Pridmore, T., Gladman, J., &

Edmans, J. (2011). Development and Evaluation of a

Mixed Reality System for Stroke Rehabilitation. In S.

Brahnam & L. Jain (Eds.), Advanced Computational

Intelligence Paradigms in Healthcare 6. Virtual

Reality in Psychotherapy, Rehabilitation, and

Assessment (pp. 193–228). Springer Berlin

Heidelberg. doi:http://dx.doi.org/10.1007/978-3-642-

17824-5_10.

Hocine, N., & Gouaich, A. (2011). Therapeutic games’

difficulty adaptation: An approach based on player's

ability and motivation. Computer Games (CGAMES),

2011 16th International Conference on.

doi:10.1109/CGAMES.2011.6000349.

Hocoma. (2013). Armeo Spring. Retrieved from

http://www.hocoma.com/products/armeo/armeospring/.

InMotion. (2013). InMotion Robots for Rehabilitation.

Retrieved from http://interactive-motion.com/.

Krakauer, J. W. (2006). Motor learning: its relevance to

stroke recovery and neurorehabilitation. Current

Opinion in Neurology, 19(1). Retrieved from

http://journals.lww.com/co-neurology/Fulltext/2006/

02000/Motor_learning__its_relevance_to_stroke_reco

very.14.aspx.

Ma, S., Varley, M., Shark, L., & Richards, J. (2010). EMG

Biofeedback Based VR System for Hand Rotation and

Grasping Rehabilitation. In Information Visualisation

(IV), 2010 14th International Conference (pp. 479–

484). doi:10.1109/IV.2010.73.

Maclean, N., Pound, P., Wolfe, C., & Rudd, A. (2000).

Qualitative analysis of stroke patients’ motivation for

rehabilitation. BMJ, 321(7268), 1051–1054.

doi:10.1136/bmj.321.7268.1051.

McKay, J., Mensah, G. A., & Greenlund, K. (2004). The

atlas of heart disease and stroke. World Health

Organization.

Murray, C. D., Pettifer, S., Howard, T., Patchick, E. L.,

Caillette, F., Kulkarni, J., & Bamford, C. (2007). The

treatment of phantom limb pain using immersive

virtual reality: Three case studies. Disability and

Rehabilitation, 29(18), 1465–1469. doi:10.1080/

09638280601107385.

Oboe, R., Daud, O. A., Masiero, S., Oscari, F., & Rosati,

G. (2010). Development of a haptic teleoperation

system for remote motor and functional evaluation of

hand in patients with neurological impairments. In

Advanced Motion Control, 2010 11th IEEE

VirtualArmRepresentationandMultimodalMonitoringfortheUpperLimbRobotAssistedTeletherapy

International Workshop on (pp. 518–523).

doi:10.1109/AMC.2010.5464078.

Parmanto, B., & Saptono, A. (2009). Telerehabilitation:

State-of-the-Art from an Informatics Perspective.

International Journal of TeleRehabilitation, 1(1).

doi:doi: 10.5195/ijt.2009.6015.

Prashun, P., Hadley, G., Gatzidis, C., & Swain, I. (2010).

Investigating the Trend of Virtual Reality-Based

Stroke Rehabilitation Systems. In Information

Visualisation (IV), 2010 14th International

Conference (pp. 641–647). doi:10.1109/IV.2010.93.

Qiu, Q., Fluet, G. G., Saleh, S., Ramirez, D., &

Adamovich, S. (2010). Robot-assisted virtual

rehabilitation (NJIT-RAVR) system for children with

cerebral palsy. In Bioengineering Conference,

Proceedings of the 2010 IEEE 36th Annual Northeast

(pp. 1–2). doi:10.1109/NEBC.2010.5458203.

Rego, P. A., Moreira, P. M., & Reis, L. P. (2010). Serious

games for rehabilitation: A survey and a classification

towards a taxonomy. In Information Systems and

Technologies (CISTI), 2010 5th Iberian Conference

on (pp. 1–6).

Reinthal, A., Szirony, K., Clark, C., Swiers, J., Kellicker,

M., & Linder, S. (2012). ENGAGE: Guided Activity-

Based Gaming in Neurorehabilitation after Stroke: A

Pilot Study. Stroke research and treatment, 2012.

Sveistrup, H. (2004). Motor rehabilitation using virtual

reality. Journal of NeuroEngineering and

Rehabilitation, 1(10). doi:10.1186/1743-0003-1-10.

World Heath Organization. (2011). Global Health and

Aging.

NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics