Illusion Approach for Upper Limb Motor Rehabilitation
Yee Mon Aung and Adel Al-Jumaily
Schoolof Electrical, Mechanical and Mechatronic Systems, Faculty of Engineering,
University of Technology Sydney, 15 Broadway, Ultimo, NSW 2007, Australia
Keywords: Upper Limb Rehabilitation, Augmented Reality Therapy and Illusion based Motor Rehabilitation.
Abstract: Taking the advantage of human brain plasticity nature, Augmented Reality (AR) based Illusion System
(ARIS) for upper limb rehabilitation has developed. The ARIS aims to restore the lost functions of upper
limb due to various motor injuries. It incorporates with AR technology to build up the upper limb
rehabilitation exercise and computer vision with color recognition technique to comply “Fool-the-Brain”
concept for fast recovery of neural impairments. The upper limb exercise that developed in ARIS is to
promote the impaired arm range of motion by moving along the predefined trajectory of the AR based
exercise. In ARIS, the real impaired arm will be overlapped by the virtual arm throughout the rehabilitation
exercise to create the illusion scene. In the case of real arm cannot perform the required task, virtual arm
will take over the job of real one and will let the user to perceive the sense that he/she is still able to perform
the reaching movement by own effort to the destination point which is the main idea of ARIS. The
validation of ARIS was conducted as a preliminary stage and the outcome are discussed.
1 INTRODUCTION
Neurotrauma such as Traumatic Brain Injury (TBI),
Spinal Cord Injury (SCI) or Stroke survivors are
facing with functional neurological deficit including
motor deficit resulting loss of control over whole
body or one side of the body in accordance with the
degree of impairment. Therefore, patients cannot
perform the daily live activities and this will greatly
impact their quality of life. According to the Physical
Disability Australia (PDA), it was estimated that
13% of the Australian population have suffered from
physical disabilities that limits the mobility and self-
care activities (Physical Disability Australia., 2013).
To improve such limitations, rehabilitation is
essential to perform. Rehabilitation is the physical
therapy that regains mobility, strength of muscles
and relearning skills. Research studies had proven
that the motor impairment can be treated by intense
use of active movement in repetitive tasks and task-
orientated activities which will result in improving
motor skills and muscular strength by preventing
muscle spasticity, muscle atrophy and osteoporosis
(Riener et al., 2005). Thus, a lot of researchers have
been trying to develop low cost effective
rehabilitation systems by employing various
technologies that including virtual reality and
augmented reality for therapeutic purposes as these
technologies have proven their effectiveness in the
area of upper limb rehabilitation (Al-Issa et al.,
2012).
This paper provides a short review of recent
developments for upper limb rehabilitation with
augmented reality technology. Following a brief
overview, the proposed novel system will be
described which is AR based Illusion System (ARIS)
for upper limb rehabilitation. The proposed system
integrates physical and psychological rehabilitation
possibilities for fast recovery in upper limb motor
deficit. In term of physical rehabilitation, AR based
reaching exercise has developed to attain the wider
range of movements at shoulder joint. In term of
psychological rehabilitation, ARIS employs “Fool-
the-Brian” concept to create the “illusion scene” to
provide “fool the sense” to the patients. This concept
will provide with artificial visual feedback to the
system user to reestablish the neural pathways and
synapses that able to control the mobility by oneself.
The term “Fool-the-Brain” was defined by the group
of researchers from (Giraux and Sirigu, 2003) as their
studies had proven that motor cortex activity was
able to restore successfully by illusory movements of
paralyzed limb and able to “fool” the brain to
improve the arm/hand movements. It is because of
the ability of human neuroplasticity which is
99
Mon Aung Y. and Al-Jumaily A..
Illusion Approach for Upper Limb Motor Rehabilitation.
DOI: 10.5220/0004647300990105
In Proceedings of the International Congress on Neurotechnology, Electronics and Informatics (VirtRehab-2013), pages 99-105
ISBN: 978-989-8565-80-8
Copyright
c
2013 SCITEPRESS (Science and Technology Publications, Lda.)
mediated by beliefs, perceptions and emotions that
react in ever-changing ways, through thoughts and
activities based on environmental input.
2 RELATED WORK
Recently, Augmented Reality (AR) technologies in
upper limb rehabilitation area provide promising
results as a useful tools for physical, psychological
and occupational therapies. AR is the combination
of real and virtual environment where virtual objects
lay on top of the real environment which is fed by
camera as a live video background image on the
computer display screen. Therefore, this approach
will provide better interactive training environment
and more realistic. As far as AR based upper limb
rehabilitation systems are concern, researchers in
(Aung et al., 2012a; Dinevan et al., 2011) have
developed AR based system to promote the reaching
movements of upper limb and strengthen the
associated muscles. Our group has previously
developed AR based upper limb rehabilitation
system which integrated with biofeedback system to
access the performance of trained muscles and
provide with more motivational approach (Aung et
al., 2012b; Aung and Al-Jumaily, 2013). In terms of
“Fool-the-Brain” concept, the motivation is arisen
from the positive results of mirror box therapy (Lin
et al., 2012) and TeraMem System (Regenbrecht et
al., 2012) which has employed “Fool-the-Brain”
concept successfully. The numbers of studies have
embarked on employing mirror therapy to enhance
motor deficit and provide with positive results (Lee
et al., 2012; Thieme et al., 2013).
However, up to our knowledge, the existing
developments based on this concept are only
developed with forearm and/or hand rehabilitation
with mirror box to attain the artificial visual
feedback. Therefore, the very first novel system that
training for the whole arm with artificial visual
feedback is proposed in this paper. The detail of the
development is described in the following section.
3 DEVELOPMENT OF “ARIS”
The AR based Illusion System (ARIS) is a low cost
rehabilitation system that only requires PC with
cheap webcam and any color as a marker.
Employing of AR technology in ARIS aims to
motivate the patients’ willpower to perform the long
term rehabilitation therapy. The ARIS is developed
in Adobe Flash Professional CS6 platform. It works
based on capturing the video scenes, displaying of
virtual objects, manipulating of 3D Virtual Arm
(VA) that is loaded with Papervision 3D, detecting
the color marker and checking for the collision
detection between marker and virtual objects. The
system architecture and the development of the
ARIS detail as below.
3.1 System Architecture
The overall architecture of the proposed system is
shown in figure 1. The upper limb rehabilitation
therapy system starts with displaying the therapy
window on the monitor that includes the information
for user, rehabilitation exercise and tracking result.
Then, the system will request the permission to
access the webcam to start the therapy. Once user
allows the webcam access, live video image will
feed at the defined position on the therapy window.
This live video will serve as a background for AR
based rehabilitation exercise. Subsequently, the user
will be requested to click on the desired color at
shoulder joint to track the current joint position and
then the shoulder joint of Virtual Arm (VA) will be
attached at clicked position. This attachment will
overlay the virtual arm on top of the real impaired
arm to perceive “fool-the-brain” concept from user.
Figure 1: System Architecture.
NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics
100
After that, the user will be requested to click again
on second color at wrist joint and this selected color
will interact with virtual objects to perform the
rehabilitation exercise. After all of the above steps
are completed, the user/therapist will have to press
the keyboard to activate the exercise and auto
movement of VA. The developed system allows the
user and therapist to monitor the performance of real
arm motion under “Real Arm Trajectory Graph” in
therapy window. The complete system is made up of
four different modules: computer vision and
multicolour tracking, modelling of virtual arm,
modelling of kinematic chain and AR based upper
limb rehabilitation exercise as depicted in figure 2.
The detail developments of each module are
explained in the following sections.
3.2 Modelling of Virtual Arm
The Virtual Arm (VA) is created in blender software
as portrayed in figure 2(b) and then exported as
“.dae” format to load in the Adobe Flash CS6 via
Papervision 3D which is an open source real-time
3D engine for Flash. In blender software, the arm
kinematic chain is defined correctly in such a way
that the upper arm as a parent link of the forearm
link, the forearm link as a parent link of the hand
link. This arrangement will ensure any motion of the
parent link will have an effect on the child link
during simulation. The defining of pivot point on
each link is very important as this will define the
centre of rotation of the link joint and is carefully
defined in developed VA. After developing VA in
blender software, the kinematic model of the VA is
developed as shown in figure 2(c). In the kinematic
model, the shoulder joint is considered as spherical
joint that permits the wide range of joint motions.
However in this paper, the shoulder joint is limited
to 3DOF, the elbow joint, forearm and wrist joint is
modelled as cylindrical joint which allows the
flexion/extension of the elbow and wrist and internal
rotation of the forearm with 1DOF each. Therefore
total of 6DOF model is developed and Denavit-
Hartenberg (D-H) parameters are assigned for
kinematics calculation as detail in next section.
3.3 Modelling of Kinematics Chain
The mathematical modeling of VA was formulated
with Denavit-Hartenberg (D-H) convention (Denavit
& Hartenberg, 1964). The D-H model was created
by relating each link’s frame of reference on the
virtual arm with the preceding link frame of
reference. The four parameters: a, , d and which
represents the link length, link twist, link offset and
joint angle, respectively are defined at respective
joint as shown in Table 1.
Table 1: D-H parameters and anatomical motion and limit.
The frame of reference is established depending on
the translation and rotation in x-axis (a and ) and
the translation and rotation in z-axis (d and ). Each
property used in D-H convention is constant, except
if the joint is a revolute type, link twist “” is
variable, or when the joint is prismatic, link offset
“d” is variable. Although the real arm has wider
articulation range, the VA is limited to safe
articulation range and the values of such range, D-H
parameters of VA and anatomical motions with
respect to joints are presented in Table 1. Forward
and inverse kinematics calculation is performed to
reach the predefined position according to the
desired trajectory that based on developed exercise.
The inverse kinematic is computed for each link
angle, following the link by link downward from
shoulder to wrist.
(a) (b) (c) (d)
Figure 2: Individual module in ARIS (a) Computer vision & multicolor tracking (b) Modeling of virtual arm (c) modeling
of kinematic chain (d) AR based rehabilitation exercise.
IllusionApproachforUpperLimbMotorRehabilitation
101
There are total in seven desired points that real
arm and VA require to move as an upper limb
rehabilitation exercise which is in circular motion.
The circular motion as a rehabilitation exercise has
been chosen and development will be explained in
the following section.
3.4 AR based Rehabilitation Exercise
Rehabilitation exercise is essential for post stroke
patients and any motor injured patients as this will
help the patients to improve their lost functions by
performing repetitive tasks and task-oriented
activities. To perform such repetitive task-oriented
movement, Circular Motion Exercise (CME) has
developed in ARIS. The trajectory of CME is chosen
as the circular motion because it involves almost
largest range of motion of the shoulder articulations
such as shoulder flexion, adduction and abduction,
internal and external rotation which is an optimum
goal to achieve as upper limb rehabilitation. CME is
built based on AR technology where virtual world
and real world are merged.
(a) (b)
Figure 3: Circular Motion Exercise (CME) for Upper
Limb Rehabilitation (a) Without video Background (b)
With video Background.
The aim of the CME is to collect the virtual objects
that indicated by animated arrows and place them on
the respective transparent virtual objects that are
displayed on the screen. The screen shot of CME is
portrayed in figure 3. There are total of seven virtual
objects that user requires to pick and place as shown
in the figure 3(a). To pick the virtual object, user
requires moving his/her arm to pick up point and
then the color that selected at wrist joint (in this case
is blue color) will check the collision detection with
virtual object. If the color and the virtual object are
collided to each other, it defines as picking of the
object and then virtual object will attach to the color
at wrist joint. This will appear on the display screen
as user is picking up the virtual object. When the
object reaches to placing position, the collision
between virtual object and predefined placing
position will check again to place the virtual object.
The place positions of all seven virtual objects are
appeared as same translucent virtual objects in
circular shape for easy understanding.
The rehabilitation purpose of the CME is to
provide the wider range of motion at shoulder joint
which is one of the most important exercises in
rehabilitation hospital. By performing CME, user
arm will train in shoulder flexion, abduction,
adduction, internal and external rotation movements.
As a result, the associated muscles that involved in
these movements will be trained and strengthen over
time.
3.5 Multicolour Tracking
In this work, only two joints: shoulder joint and
wrist joint position are interested to be tracked as the
current position of that particular joint. The shoulder
joint position is required to overlay the VA so that
wherever the real shoulder joint is captured by
webcam, the VA will always overlay correctly on
top of the real arm and display on the monitor. As
for wrist joint position, this is where interacting with
real hand and virtual object collision is occurred.
Figure 4: Four layer to built up "Fool-the-Brain" concept.
Another purpose of the wrist joint tracking is to
track the performance of the real hand trajectory that
user and therapist are able to monitor. This
performance can be observed in real-time under
“Real Arm Trajectory Graph” in the therapy
window. Therefore, we have developed multicolor
tracking algorithm that the system is able to track the
multiple colors properly with minimum effect due to
the surrounding light changes.
NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics
102
4 “FOOL-THE-BRAIN”
CONCEPT
In ARIS, “Fool-the-Brain” concept is developed via
overlaying of live video image and virtual objects
with multiple layers to create the illusion of user’s
perception. The idea of overlaying concept is as
portrayed in figure 4. There are four layers in total to
achieve the concept. The very first layer is fed with
live video image to create AR environment. On top
of that layer, rectangular shape with rounded edges
object is overlaid which defined as second layer. In
this layer, webcam captured image is imported.
However, this imported image can be only seen
inside of the rectangular shape object. The rest of the
region (out of rectangle shape) is set as transparent
so that the active region is only inside the
rectangular shape object. By overlaying the
rectangular object, this will appear as covering the
actual human arm on the display screen and will
appear as removing of real arm on the display
screen. The placement of the rectangular shape
object is attached to the user defined color at
shoulder joint. Therefore, current position of the real
arm will be always tracked and overlapped with
user’s arm size image which captured via webcam
wherever the user’s arm moves. Then the CME will
overlay at third layer in ARIS to serve as an upper
limb rehabilitation exercise. At fourth layer, VA will
be overlaid on top of the real arm and appear as it is
covering on top of rectangular shape object.
Therefore, this appearance will create the illusion as
a user real arm. This means that the real arm of the
user will be removed and replaced with VA by
means of computer vision and color tracking
technique. Therefore, from the user point of view,
he/she will perceive the VA as his/her own arm.
When CME start, both VA and real arm will move
together to reach the desired position where the real
arm is moved by user effort and VA will move by
predefined trajectory via inverse kinematic. In the
case of real arm cannot reach to the desired position,
VA will take over the job of real arm as if the user
arm is still moving and reach to the desired position
while the real arm is covered up with rectangular
shape object with same background of live webcam
so that user will not see his/her real arm. This will
let the user to perceive the sense that he/she is still
able to perform the reaching exercise by his/her own
impaired arm to the destination point. This sense
will help the patients in fast recovery of motor
impairment due to human brain plasticity.
5 PROOF OF CONCEPT STUDY
Non-clinical study has tested for novelty aspects of
overlaying or illusion, augmentation and
believability. In this work, a healthy subject (user) is
requested to sit in front of the computer monitor and
webcam. After that, ARIS program will run and the
therapy window is appeared on the display screen.
The instruction on how to perform the rehabilitation
exercise and its step by step procedure is stated at
the right bottom of the therapy window. The current
X, Y, Z positions and joint angle values of shoulder,
elbow and wrist joints of the VA is able to monitor
by the user in real-time on the therapy window.
Before the rehabilitation exercise starts, the request
command for the webcam is appeared. After user
has accepted the request, seven translucent virtual
objects are appeared which displayed in circular
shape on top of the real-time video background
image. Then the user is asked to select the color at
shoulder joint (in this case is green). Once the color
has chosen at shoulder joint, VA is appeared on top
of the real arm that capture by webcam. The VA that
overlay on top of the real arm is moved and tracked
correctly according to the real arm movement. This
shows that the developed tracking algorithm is able
to track quite accurately and less response to light
changes. Once VA is overlapped, the user is asked
again to choose another color at the real wrist joint
that captured by webcam (in this case is blue). Then,
the system calculates the distance between the
shoulder joint and wrist joint to calculate the length
of rectangular shape object. Subsequently, the
webcam captured image is cover up on top of the
real arm. After that the selected color is interacted
with seven virtual objects to perform pick and place
action. We observed that the real arm is successfully
covered up and picking and placing of virtual
objects are able to perform smoothly. The collision
detection method that checking the pixels distance
between wrist color marker and virtual object is able
to detect precisely. Throughout the performing of
rehabilitation exercise, VA is able to move
according to the predefined trajectory which is
driven by forward and inverse kinematic algorithm.
The desired position and joints angles are able to be
displayed correctly on the therapy window as well.
The testing of overtaking task has also conducted
where if there is a case that the real arm cannot
move to the desired position. It has found that VA is
successfully continued to carry out the task by
placing the virtual objects at desired positions. This
approves that “Fool-the-Brain” concept is
successfully adopted in this development. The
IllusionApproachforUpperLimbMotorRehabilitation
103
tracking of real hand movement is able to monitor in
real-time under “Real Arm Trajectory Graph”
effectively. The screen shot of the proof of study is
depicted in figure 5.
Figure 5: Result from proof of concept study.
6 CONCLUSIONS AND FUTURE
WORK
In this paper, ARIS was developed to improve the
upper limb range of motion especially for shoulder
joint motion with “Fool-the-Brain” concept. The AR
based computer game is chosen as a rehabilitation
exercise as integrating with AR technology in
rehabilitation field provides high motivation for long
term training. Therefore, AR based CME was
developed in ARIS to attain the wide range of
shoulder joint motions via reaching exercise.
Throughout the performing of CME rehabilitation
exercise, the real arm was overlapped by VA model
to fool the user perception to attain artificial visual
feedback that provides fast recovery of user’s neural
impairment. If the user cannot reach to the particular
point, VA will take over the job of real arm and will
let the user to perceive the sense that he/she is still
able to perform the reaching movement by own
effort to that particular point. Non-clinical study was
conducted to validate the novelty aspects of ARIS
and provided with positive observations to carry on
with future works.
As far as future work is concerned, improvement
of the appearance of the VA model will be made.
The model will be replaced with user skin texture
that can be adjusted according to the individual user
complexion. This will perceive the more realistic
appearance of virtual arm as a real user arm. To
make it more effective rehabilitation system, the
system will be integrated with biofeedback system to
detect the intended movement of the user via surface
eletromygraphy (sEMG) signals. Non-clinical trials
with more participants will be conducted in near
future to attain the improvement on illusion,
perception and understanding the level of interest of
developed rehabilitation exercise. Then, the clinical
trial to be conducted at Port Kembla Rehabilitation
Hospital in Australia after all the necessary
improvements have made and result will be reported
soon.
REFERENCES
Al-Issa, H., Regenbrecht, H., & Hale, L. (2012).
Augmented reality applications in rehabilitation to
improve physical outcomes. Physical Therapy
Reviews, 17(1), 16-28.
Aung, Y. M., & Al-Jumaily, A. (2012a, 24-27 June 2012).
AR based upper limb rehabilitation system. Paper
presented at the 4th IEEE RAS & EMBS International
Conference on Biomedical Robotics and
Biomechatronics (BioRob), Rome, Italy.
Aung, Y. M., & Al-Jumaily, A. (2012b, 5-8 Aug. 2012).
Shoulder rehabilitation with biofeedback simulation.
Paper presented at the 2012 International Conference
on Mechatronics and Automation (ICMA), Chengdu,
China.
Aung, Y. M., & Al-Jumaily, A. (2013). Neuromotor
Rehabilitation System with Real-Time Biofeedback.
International Journal of Computer Information
Systems and Industrial Management Applications, 5,
550-556.
Denavit, J., & Hartenberg, R. S. (1964). Kinematic
Synthesis of Linkages. New York: McGraw-Hill.
Dinevan, A., Aung, Y. M., & Al-Jumaily, A. (2011, 5-8
Dec. 2011). Human computer interactive system for
fast recovery based stroke rehabilitation. Paper
presented at the 11th International Conference on
Hybrid Intelligent Systems (HIS).
Giraux, P., & Sirigu, A. (2003). Illusory movements of the
paralyzed limb restore motor cortex activity.
NeuroImage, 20, Supplement 1(0), S107-S111.
Lee, M. M., Cho, H.-y., & Song, C. H. (2012). The Mirror
Therapy Program Enhances Upper-Limb Motor
Recovery and Motor Function in Acute Stroke
Patients. American Journal of Physical Medicine &
Rehabilitation, 91(8), 689-700.
Lin, K. C., Chen, Y. T., Huang, P. C., Wu, C. Y., Huang,
W. L., Yang, H. W., . . . Lu, H. J. (2012). Effect of
mirror therapy combined with somatosensory
stimulation on motor recovery and daily function in
stroke patients: A pilot study. Journal of the Formosan
Medical Association(0).
Physical Disability Australia. (2013). What is physical
disability? Retrieved May, 2013, from
http://www.pda.org.au/what-is-physical-disability.
Regenbrecht, H., Hoermann, S., McGregor, G., Dixon, B.,
Franz, E., Ott, C., Hoermann, J. (2012). Visual
NEUROTECHNIX2013-InternationalCongressonNeurotechnology,ElectronicsandInformatics
104
manipulations for motor rehabilitation. Computers &
Graphics, 36(7), 819-834.
Riener, R., Frey, M., Bernhardt, M., Nef, T., & Colombo,
G. (2005, 28 June-1 July 2005). Human-centered
rehabilitation robotics. Paper presented at the
Rehabilitation Robotics, 2005. ICORR 2005. 9th
International Conference on.
Thieme, H., Mehrholz, J., Pohl, M., Behrens, J., & Dohle,
C. (2013). Mirror therapy for improving motor
function after stroke. Stroke, 44(1), e1-e2.
IllusionApproachforUpperLimbMotorRehabilitation
105
