AUGMENTED REALITY BASED REHABILITATION SYSTEM

WITH SELF-DESIGNED DATA-GLOVE

S. K. Ong, Y. Shen and A. Y. C. Nee

Mechanical Engineering Department, Faculty of Engineering, National University of Singapore, Singapore, Singapore

Keywords: Human-computer Interaction, Augmented Reality, Data-glove, Rehabilitation.

Abstract: In this research, a rehabilitation system based on Augmented Reality has been developed to recover hand

motions. With the self-designed data-gloves and Augmented Reality technology, intuitive human-computer

interaction has been developed to provide entertaining and engaging rehabilitation programs. A self-

designed data-glove is used to detect the grasping movements of the patient’s hand, collect the physical

information of the patient and provide tactile feedbacks according to the performance of the patient. With

the advantages of seamless merging of virtual and real elements, controllable environment and intuitive

interface, Augmented Reality based rehabilitation systems can provide entertaining exercising programs to

the patients. In the developed rehabilitation system, multimodal feedbacks are provided to reduce the fatigue

and make the patients more engaged in the rehabilitation sessions.

1 INTRODUCTION

It is expected that the proportion of people aged

above 60 years old would reach 19% by 2050, and

there would be a growing proportion of individuals

who suffer from disabilities due to illnesses related

to ageing. Rehabilitation plays a huge part in the

recovering process to maintain and restore the

maximum movements and functional ability of the

body throughout the lifespan of the individuals.

With effective rehabilitation systems, motor

functions can be recovered, the independent living

ability can be enhanced and the cost in healthcare

can be reduced. There are three key factors in motor

functions recovery, namely, early intervention, task-

oriented training and repetition intensity (Malouin et

al., 2003). Therefore, an effective rehabilitation

system should be able to provide repetitive training

with affordable costs and easy monitoring of the

recovering progress with set goals.

Although conventional rehabilitation systems

have been proven to be useful in recovering of the

motor functions (Kwakkel et al., 1999), they have

disadvantages, such as boring content, high cost and

lack of digital monitoring. New technologies can

provide a novel paradigm for therapy methods to

address these issues. Augmented Reality (AR)

technology (Azuma, 1997) augments a user’s view

of the real world by superimposing virtual objects

with the physical world. With an AR-based

interface, the users can interact with the virtual

objects in a real environment.

In this paper, a rehabilitation system for hand

movements based on AR technology, sensors and

video games is presented. A sensor-based data-glove

has been developed to detect the grasping

movements and provide tactile feedbacks so as to

support intuitive human-computer interaction

allowing patients to interact with virtual objects in a

real environment using natural hand movements.

Virtual elements which can be interacted using the

data-gloves are designed in the video games based

on AR technology to provide the entertaining

exercising environment.

2 RESEARCH BACKGROUND

Traditional rehabilitation processes to recover motor

functions are usually carried out with external

devices, e.g., the nine-hole pegboards, the exercise

hand skate, and repetitive practices. In these

systems, the exercises are closely monitored by the

therapists. A limitation in these traditional

rehabilitation systems is boredom, which is usually

observed in rehabilitation programs requiring

Shen Y., Ong S. and Nee A..

AUGMENTED REALITY BASED REHABILITATION SYSTEM WITH SELF-DESIGNED DATA-GLOVE.

DOI: 10.5220/0003806100730078

In Proceedings of the 2nd International Conference on Pervasive Embedded Computing and Communication Systems (PECCS-2012), pages 73-78

ISBN: 978-989-8565-00-6

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

intensive practice. Tedious practices without fun and

entertainment tend to reduce the motivation of the

patients. Another limitation is the high cost of these

rehabilitation systems, which would impose

considerable financial burden on the patients and

make it difficult for them to undergo the practices

needed to effect neural and functional changes.

There is no data collection function in these systems,

as conventional rehabilitation systems are designed

without sensors such that digital information of the

performance progress cannot be collected.

Providing an enhanced environment by

augmenting virtual objects into the physical world

seamlessly, AR technology can provide an intuitive

interface to rehabilitation systems. Luo et al.,

(2005a) developed rehabilitation system integrating

AR and assistive devices for hand opening actions of

stroke survivors. Dynamic feedbacks of the subject

performance are provided. Luo et al., (2005b) also

designed a rehabilitation system to facilitate the

rehabilitation of the grasp-and-release tasks, where

users without muscle strength can move virtual

objects without difficulties.

Sucar et al., (2008) developed a web-based

gesture therapy system using AR to provide

intensive motion training at low cost. Visual

feedback of the patient’s performance is provided

and the practice progress is illustrated through a

simple statistical chart. AR-REHAB (Alamri et al.,

2010) is an AR rehabilitation system to increase the

immersive feeling of the post-stroke patients.

Fiducial markers are used to track the hand

movements of the patients, which are analysed to

evaluate and monitor the performance of the

patients. AR-based games (Burke et al., 2010) have

been reported to support upper limb rehabilitation,

where AR markers are used to augment everyday

objects and track the movements of these objects.

This project takes into consideration the principles in

game designs to enhance the user engagement.

Wang et al., (2010) developed an AR system

with the use of air pressure detecting devices. Based

on the physical condition of the users, which is

measured using pressure sensing devices, the

parameters of the game are adjusted to ensure that

patients with different extents of hand impairments

can interact with the game. Correa et al., (2009)

created an AR system to facilitate accessing music

by the disabled patients, where patients without

muscle strength can use this system. The AR Piano

Tutor involves virtual objects augmented onto a real

MIDI keyboard in a video captured scene which

represents the user’s viewpoint (Barakonyi and

Schmalstieg, 2005). However, a patient with limited

finger strength and control may not be able to

depress the physical keys. Zhang et al. (2010)

projected a virtual piano onto the real environment

and a self-designed data-glove with flex sensors is

used to measure the flexing angles of the fingers.

3 METHODOLOGIES

3.1 System Architecture

The proposed system shown in Figure 1 aims to help

patients to improve their grasping strength as well as

train their hand-eye coordination via AR-based

game play with a self-designed data-glove. The

system hardware includes a standard computer, an

off-the-shelf webcam and a self-designed data-

glove. With the force sensitive resistors sewn on the

glove, the data-glove can detect the grasping force

applied on the fingers. The multiple threading

method and the function of the playlist in the

OpenGL library are used to ensure that the system

would work in real time without any time lag.

Patient

Motion Capture

Visual

Feedbac

Audio Feedback

Data-glove

Camer

Video Game

Haptic

Feedbac

Figure 1: The system architecture.

A “burst the balloon” game has been developed

to train the grasping movements of the patient’s

hand. The virtual balloons are rendered in the real

environment by tracking a marker using ARToolKit

(ARToolKit, 2002). With another marker on the

data-glove, the virtual hand overlaid on the real hand

would interact with the virtual balloons. The

collision between the virtual balloon and the

patient’s hand is determined by detecting the

collision between the virtual balloon and the virtual

hand using the VCollide library.

Upon collision between the virtual balloon and

the virtual hand, the balloon will burst when a

sufficient amount of force is applied. Performance

will be based on the number of balloons burst out of

a total of twenty balloons. There are also different

PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems

levels of difficulties that the user can choose to suit

their rehabilitation progress. With a higher difficulty

level, balloons will appear at faster speeds and a

greater force will be needed to burst the balloons.

Multimodal feedbacks, namely, visual, audio and

tactile feedbacks, are provided based on the

performance of the patients. These feedbacks are

important performance indicators. They can further

encourage the practice and maintain the interest

level of the patients. The feedbacks are provided

mainly in two aspects, viz., the change of the context

in the gaming environment and the display of the

quantitative information of the performance.

The scoring module in the system is designed to

record the accuracy of the patient interaction with

the system. Rendering the quantitative information

recorded by the scoring module onto the screen can

provide the patients with performance feedbacks.

The quantitative information can be stored, making

it convenient for the therapists to monitor the

recovering progress of the patients and analyse the

effectiveness of the rehabilitation system.

3.2 Self-designed Data-glove

The data-glove shown in Figure 2 is used to detect

the grasping forces of the fingers as the fingers

interact with the virtual objects in the AR

environment. The touching sense is applied to the

fingers with a real ball held in the user’s hand, as

shown in Figure 3. The force sensing resistors (FSR)

are sewn onto the glove to obtain the pressing forces

of the fingers. The grasping movements detected by

this data-glove can be transmitted wirelessly via

Bluetooth to the system.

The data-glove is designed to be used in the AR-

based rehabilitation system. Therefore, it should be

able to be worn on patients whose fingers cannot

move freely. Open-design is implemented to develop

the data-glove in which Velcro belts are used to affix

the data-glove to the hand, as shown in Figure 2.

This data-glove consists of five parts, i.e., the

FSR module, the computing module, the information

transmission module, the power supply module and

the feedback module. The FSR module is used to

detect the pressing forces of the fingers. The

computing module, i.e., the microcontroller

(Arduino Pro mini microcontroller) is used to collect

the analogue signals from the sensors. The

microcontroller also performs some preliminary

information processing, such as converting the

analogue signals into digital signals and basic

computation. The information transmission module

is based on the Bluetooth technology to achieve

wireless information communication between the

computer and the microcontroller. The power supply

module includes a battery and a voltage regulator to

ensure the microcontroller works under a stable

voltage of 3.3V and the Bluetooth works under a

voltage range of 3.3V~6V. For the feedback module,

miniature vibration motors are mounted on each

finger tip to provide tactile feedback when the hand

is squeezing to burst the balloon.

Figure 2: Self-designed data-glove.

Figure 3: A ball is held in the game.

When a larger pressure is applied to a FSR on the

data-glove, the resistance of the FSR will be lower.

In the circuit of the FSR, the microcontroller

provides the power supply. The current in the circuit

and the voltage between the two points of the

resistor will be changed accordingly. The voltage is

input into an analogue port of the microcontroller

and converted to a digital value through the Analog-

to-Digital converter in the microcontroller.

Calibrations have been conducted to obtain the

relationship between the voltage output in the circuit

and the force applied to the FSR. Due to the

limitation of the FSRs, the relation between the force

applied and the change in the resistance is not

directly proportional. In addition, there is only a

limited useful operating range. After a series of

calibrations, it is observed that all the FSRs on

different fingers exhibit similar trends; therefore,

only the limited range of 0N to 6N is used.

The self-designed data-glove is low cost and

affordable to normal users. It is not only a great tool

AUGMENTED REALITY BASED REHABILITATION SYSTEM WITH SELF-DESIGNED DATA-GLOVE

to improve on the user’s condition, but at the same

time monitor the patient’s progress throughout the

whole gaming process.

3.3 Burst the Balloon Interactions

The Burst the Balloon game is a simple and intuitive

game that does not require complicated instructions

and has a relatively short learning curve. When there

are virtual balloons flying from the marker into the

3D space, the patient is required to move his hand

with the data-glove to the position of the virtual

balloon and squeeze the real ball at the same time.

Therefore, the rehabilitation system can help patients

improve their grasping strength as well as train their

hand eye coordination ability.

Interaction between the virtual hand and the

virtual balloons consists of two parts, i.e., coordinate

transformation and collision detection. Since the

virtual hand and the virtual balloons are rendered in

two different coordinate systems on two markers,

coordinate transformation is needed to compare the

relative positions of the two virtual objects, using the

camera coordinate system as a bridge between these

two different markers (Figure 4). The “Hiro” marker

associated with the virtual balloons is set as the base

coordinate system. The coordinates of the “Hiro”

and the “Kanji” markers in the camera coordinate

system can be determined through analysing the

images captured by the camera using the functions

provided by ARToolKit. When signals from the

FSRs show the fingers grasping, the position (i.e., x-

, y- and z-coordinates) of the virtual hand in the

“Kanji” marker coordinate will be transformed to a

position in the “Hiro” marker coordinate system, and

compared with the position of the virtual balloon

using the VCollide library. If the virtual hand

collided with the virtual balloon, it is assumed that

the balloon is about to burst. After a collision has

been detected, a balloon bursting sound is played,

and the colour of the virtual balloon is changed from

red to green, and the miniature vibration motors

mounted on the finger tips are activated to provide

tactile feedback. In this rehabilitation system, a real

ball is held by the patient to provide more realistic

touching sense and provide haptic feedbacks when

the patient is pressing the ball.

4 IMPLEMENTATION

The rehabilitation system has been implemented on

a P4 3 GHz PC equipped with 1 GB RAM, an

ordinary web camera and a display device. The self-

designed data-glove is used as the input device.

The rehabilitation system aims to allow the user

to have a more interactive interface, with different

levels of difficulty to choose from to provide

suitable challenges during the game playing process,

which needs to be considered when the patient’s

physical conditions have been improved. Therefore,

the user interface should be designed to be

interactive, user friendly, suitable for all ages and

have different levels of difficulty.

Figure 4: Coordinate transformation between two markers.

4.1 Preparation Stage

The data-glove is simple in design and ergonomic

for use. The instructions to put on the data-glove are

as follows: (a) Strap on the main Velcro across the

palm of one’s hand, as shown in Figure 5, (b) Slot in

the respective finger into the main finger

compartment, (c) Secure the finger compartment by

strapping on the white Velcro just below the finger

compartment to the respective finger, as shown in

Figure 6, and (d) Secure the rest of the four fingers.

4.2 User Interface

In order to facilitate usability, the user interface of

the rehabilitation system is designed to give the user

real time feedback on his/her current game play. The

interface consists of the game title, score, level of

difficulty and balloons remaining in the current

exercising session, as shown in Figure 7.

Three different difficulty levels of the game,

namely, easy, normal and hard, have been developed

to cater to different capabilities of the users. They

can be selected by pressing the ‘E’, ‘N’, ‘H’ key on

the keyboard respectively. The parameters that are

different between the levels of difficulty consist of

the flying speed of the balloons and the amount of

PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems

force to burst the balloons, both of which increase as

the level of difficulty increases.

Figure 5: Put-on the data-glove.

Figure 6: Secure the finger compartment.

Figure 7: Interface of the rehabilitation system.

For each level of difficulty, twenty virtual

balloons will appear from the marker. The user

needs to move his hand to the position of the virtual

balloon and squeeze it, as shown in Figure 8. If the

pressure applied is larger than the force defined for

the selected level of difficulty, the balloon will burst,

playing a “pop” sound followed by vibrations from

the miniature motors. There will be a visual cue

where the balloon will turn from red to green, as

shown in Figure 9. For every successful action, the

score will be increased by 1. The final score will be

rendered to indicate the performance. To restart the

game, the user can select the level of difficulty using

the ‘E’, ‘N’, ‘H’ key on the keyboard, after which

the score and the balloon count will be reset.

Figure 8: A virtual balloon is rendered in 3D space.

Figure 9: The colour of the balloon becomes green.

4.3 Discussion

Comparing the developed system with existing

systems, the AR-based system presented in this

paper has a number of advantages. The data-glove is

low cost (SGD$150) as compared to commercial

force measurement gloves, such as the CyberGrasp

system with a cost of SGD$39000. The scoring

module as well as the multimodal feedbacks can be

monitored and indicate the patient’s progress easily.

The learning curve of the rehabilitation system is

short. This avoids the hassle of explaining the

instructions to the patients especially the elderly or

mentally challenged patients. The system activates

three main senses, namely visual, touch and hearing,

of the users during the exercising processes.

There are a few disadvantages of this proposed

system. The data-glove has limited force sensing

capability, which is only up to 6N. The accuracy is

also limited due to the natural property of the FSR.

More advanced sensors can be implemented to

overcome these limitations. Similar to other AR

games, lighting and visibility of the markers would

AUGMENTED REALITY BASED REHABILITATION SYSTEM WITH SELF-DESIGNED DATA-GLOVE

affect the rendering of the virtual objects.

5 CONCLUSIONS

In this research, a rehabilitation system based on AR

technology and sensors has been developed to

recover the grasping movements and the ability of

hand-eye coordination of the patients. The first

contribution of this research is the self-designed

data-glove to measure the grasping forces applied to

the fingertips. With this low-cost data-glove, the

grasping forces and the physical condition of the

patient can be estimated. The second contribution is

the integration of this data-glove with the AR-based

rehabilitation application to provide haptic, audio

and visual feedbacks to the patients using a game

play session. With a real ball held in hand, the

patient can receive haptic feedback, while the virtual

balloon rendered using AR technology can enhance

the effectiveness of the rehabilitation system in

engaging and entertaining the patients and facilitate

the training of the hand-eye coordination of the

patient. With this low-cost rehabilitation system, a

patient can take intensive exercises at home.

Future work will include developing more levels

of difficulty to make the rehabilitation system

suitable for patients with different physical

conditions. The developed system will be evaluated

with a set of criteria. A user study with a large

sample will be conducted to verify the effectiveness

of the developed system and collect more

quantitative information about the system

performance to determine how this rehabilitation

system can support and complement the

rehabilitation process of the users.

ACKNOWLEDGEMENTS

The authors acknowledge contributions from Toh

Wei-de.

REFERENCES

Alamri, A., Cha, J. and Saddik, A. E., 2010. AR-REHAB:

An augmented reality framework for post stroke-

patient rehabilitation. IEEE Transactions on

Instrumentation and Measurement, 59(10), 2010, pp.

2554-2563.

ARToolKit, 2002. ARToolKit2.61 Software. Retrieved 21

September 2011, from: http://www.hitl.washington.

edu/research/shared_space/download/.

Azuma, R., 1997. A Survey of Augmented Reality.

Presence: Teleoperators and Virtual Environments,

6(4), 355 – 385.

Barakonyi, I. and Schmalstieg, D., 2005. Augmented

reality agents in the development pipeline of computer

entertainment. In Proceedings of the 4th International

Conference on Entertainment Computer, Sanda,

Japan, 2005 September 19-21, pp. 345-356.

Burke, J. W., McNeill, M. D. J., Charles, D. K., Morrow,

P. J., Crosbie, J. H. and McDonough, S. M., 2010.

Augmented reality games for upper-limb stroke

rehabilitation. Second International Conference on

Games and Virtual Worlds for Serious Applications,

2010, pp. 75-78.

Correa, A. G. D., Ficheman, I. K., Nascimento, M. do and

Roseli de Deus, L., 2009. Computer Assisted Music

Therapy: a Case Study of an Augmented Reality

Musical System for Children with Cerebral Palsy

Rehabilitation. 2009 Ninth IEEE International

Conference on Advanced Learning Technologies, New

York, pp. 218-220.

Kwakkel, G., Wagenaar, R. C., Twisk, J. W., Lankhorst G.

J. and Koetsier, J. C., 1999. Intensity of leg and arm

training after primary middle-cerebral artery stroke: a

randomised trial. Lancet, 354(9174), 191-196.

Luo, X., Kline, T., Fischer, H. C., Stubblefield, K. A.,

Kenyon, R. V. and Kamper, D. G., 2005a. Integration

of Augmented Reality and Assistive Devices for Post-

Stroke Hand Opening Rehabilitation. In Proceedings

of the 2005 IEEE, Engineering in Medicine and

Biology 27th Annual Conference, Shanghai, China,

September 1-4, 2005, pp. 6855-6858.

Luo, X., Kenyon, R. V., Kline, T., Waldinger H. C. And

Kamper, D. G., 2005b. An augmented reality training

environment for post-stroke finger extension

rehabilitation. IEEE the 9th International Conference

on Rehabilitation Robotics. New York, 2005, pp. 329-

332.

Malouin, F., Richards, C. L., McFadyen, B. and Doyon, J.,

2003. New Perspectives of locomotor rehabilitation

after stroke. Médecine Sciences (Paris), 19(10), 994-

998.

Sucar, L. E., Leder, R. S., Reinkensmeyer, D., Hernández,

J., Azcárate, G., Casteñeda, N. and Saucedo, P., 2008.

Gesture Therapy - A Low-Cost Vision-Based System

for Rehabilitation after Stroke. HEALTHINF, Funchal,

Madeira, Portugal, January 28-31, 2008, pp.107-111.

Wang, H., Hsu, C., Chiu, D. and Tsai, S., 2010. Using

augmented reality gaming system to enhance hand

rehabilitation. 2nd International Conference on

Education Technology and Computer, Shanghai,

China, 22-24 June 2010, V3-243-V3-246.

Zhang, D., Shen, Y., Ong, S. K. and Nee, A. Y. C., 2010.

An Affordable Augmented Reality Based

Rehabilitation System for Hand Motions.

International Conference on CYBERWORLDS, 20-22

October 2010, Nanyang Technological University,

Singapore.

PECCS 2012 - International Conference on Pervasive and Embedded Computing and Communication Systems