Performance Analysis for Upper Limb Rehabilitation in Non-Immersive

and Immersive Scenarios

Vanesa Herrera

1 a

, Ana Reyes-Guzm

2 b

, David Vallejo

1 c

, Jos

e Castro-Schez

1 d

Dorothy N. Monekosso

3 e

, Gonz

alez-Morcillo Carlos

1 f

and Javier Albusac

1 g

School of Computer Science, University of Castilla-La Mancha, Ciudad Real, Spain

Hospital Nacional de Parapl

ejicos de Toledo, Toledo, Spain

Department of Computer Science, University of Durham, Durham, County Durham, U.K.

Keywords:

Virtual Reality, Immersive Environments, Upper Limb Rehabilitation, Hand Tracking, Free-Hand Interaction.

Abstract:

In recent years, new technologies have contributed to an improvement in the upper limb rehabilitation pro-

cess as a complement to the conventional therapy received by patients. In this context, technologies should

facilitate accurate monitoring of the hands and serve to collect data on patient performance during functional

tasks in order to objectively assess the patient’s potential evolution. Mechanical and wearable devices provide

greater accuracy in measurements. However, the physical limitations of patients requires the use of hands-free

solutions. This article investigates controller-free hand technologies for accurate hand tracking in the Box and

Block test (BBT) benchmarked against the real test, validated and standardized in the context of the Hospital

Nacional de Parapl

ejicos (Toledo, Spain). In particular, the performance in the execution of therapeutic ex-

ercises is analyzed in three different scenarios: i) physical environment without the use of technologies, ii)

non-immersive virtual environment and, ﬁnally, iii) fully immersive virtual environment. The results demon-

strate the similarity between therapies developed in physical scenarios without the use of technologies, and

those carried out in virtual reality-based scenarios.

1 INTRODUCTION

The interdisciplinary use of new technologies has fa-

vored their expansion into areas not originally de-

signed for. Robotics as an aid to physical movement

(Mekki et al., 2018), robots that interact with the pa-

tient, wearable devices (Bravo and Mu

noz, 2022), se-

rious games (Pereira et al., 2020), and even the com-

bination of several of them (Guill

en-Climent et al.,

2021) has become an increasingly widespread prac-

tice to help improve patients with different physical

or psychological pathologies. A clear example is the

case of video games and virtual reality, whose pur-

pose originally intended for entertainment, has re-

https://orcid.org/0000-0002-6187-4794

https://orcid.org/0000-0003-2905-2405

https://orcid.org/0000-0002-6001-7192

https://orcid.org/0000-0002-0201-7653

https://orcid.org/0000-0001-7322-5911

https://orcid.org/0000-0002-8568-9542

https://orcid.org/0000-0003-1889-3065

sulted in great interest in ﬁelds such as education,

health, or digital marketing. In particular, the use

of virtual reality in medicine and rehabilitation is

increasingly employed in hospitals and other health

centers. One main reason is the results obtained

through non-immersive, semi-immersive, and immer-

sive applications in patients affected by a wide range

of problems: neurological problems (Lamash et al.,

2017), eating disorders (Clus et al., 2018), phobias,

or spinal cord injuries (de Ara

ujo et al., 2019), among

others. It is the case of the Hospital Nacional de

Parapl

ejicos de Toledo (Spain)

, a center specializing

in patients with spinal cord injuries. The hospital em-

ploys an interdisciplinary team to evaluate, analyze,

intervene, and guide the patient to achieve the highest

degree of recovery and independence.

Spinal cord injury (SCI) affects conduction of sen-

sory and motor signals across the site(s) of lesion(s),

as well as the autonomic nervous system (Rupp et al.,

2021). This type of injury can be caused by trauma or

https://hnparaplejicos.sanidad.castillalamancha.es/

Herrera, V., Reyes-Guzmán, A., Vallejo, D., Castro-Schez, J., Monekosso, D., González-Morcillo, C. and Albusac, J.

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios.

DOI: 10.5220/0011711800003467

In Proceedings of the 25th International Conference on Enterprise Information Systems (ICEIS 2023) - Volume 2, pages 231-242

ISBN: 978-989-758-648-4; ISSN: 2184-4992

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

231

medical conditions such as degenerative diseases, tu-

mors, or infections. The incidence rate of SCI patients

has been increasing and ranges from 13,019 per mil-

lion to 163,420 per million people worldwide (Kang

et al., 2017). Most patients with SCI have impaired

function of the upper limbs (Wyndaele and Wyndaele,

2006). In this context, one of the most common con-

sequences after injury is loss of muscle strength and

numbness, increased muscle tone (spasticity) (Hod-

kin et al., 2018), weakness or paralysis. Thus, reha-

bilitation is essential to improve muscle strength and

hand/arm function in both early and chronic stages.

Rehabilitation aims to improve the patient’s quality

of life so that he or she can perform activities of daily

living autonomously or at least achieve the highest de-

gree of autonomy possible (Spooren et al., 2011).

Speciﬁcally, occupational therapy focuses on the

patient’s ability to cope with most everyday tasks.

A team of functional rehabilitation specialist, phys-

iotherapists, and other specialists plan treatment tai-

lored to the needs of each patient. Rehabilitation be-

gins at an early stage and can last for life, so it is

important to maintain the patient’s motivation and in-

volvement. The repetition of inadequate movements

has a negative impact on recovery, and it is necessary

to prevent the patient from performing compensatory

movements with other parts of the body. Furthermore,

the repetition of compensatory movements prevents

strengthening the area to be treated.

In accordance with the above problems, new tech-

nologies can help capture each patient’s kinematics

with the dual objective measuring movements and

evaluation of the execution of each exercise. There

are different alternatives for motion capture: 2D or 3D

cameras, magnetic sensors, mechanical skeletons, and

inertial sensors, among others. Some systems provide

complete solutions, such as virtual reality headsets.

They are Head Mounted Display (HMD) combined

with IMUs (Inertial Mesure Unit) that allow the user

to immerse in virtual worlds and interact with the el-

ements of the 3D environment. Some of the best-

known are Oculus Go, Oculus Rift, Oculus Quest,

Oculus Quest 2, HTC Vive Focus, HTC Vive Pro 2,

HTC Vive Focus 3, among others. In this type of de-

vice, the main form of interaction is via controllers.

However, some of these HMDs, thanks to the cam-

eras and sensors they integrate, are beginning to pro-

vide libraries that allow the development of applica-

tions. The main form of interaction is made through

the movement and gestures of the hands. This is the

case of Meta Quest 2 VR headset.

However, not all VR-based technologies are suit-

able for use in this type of patient. The use of ele-

ments such as controllers limits the functional move-

ments and grasps that can be performed by humans

(Everard et al., 2022). In addition, virtual appli-

cations with controllers cannot be used by a large

number of patients with SCI with affecting the upper

limbs. These patients generally, do not have sufﬁcient

strength or mobility. Also the use of controllers do not

allow the natural movements and grasps of the hand

to be performed freely. These movements include:

ﬂexion-extension of the wrist, ﬂexion-extension of the

index ﬁnger, gripping ability, hand and arm move-

ments and end-terminal pinches.

There are different tests to assess the grasp func-

tion of the hand. In rehabilitation and occupational

therapy, the Box and Block test (BBT) is widely used

because it allows unilaterally measuring the degree of

manual dexterity (Oliveira et al., 2016).

This article uses the BBT in three different sce-

narios. First, a physical scenario without the use

of technologies. Secondly, a non-immersive virtual

environment in which the Leap Motion Controller

(LMC) sensor is used for hand tracking and a screen

to display hands and objects virtually. Finally, the

last scenario is fully immersive and uses the Oculus

Quest 2. Virtual reality scenarios are tested by thera-

pists and clinical staff in charge of rehabilitation ses-

sions for patients with spinal cord injuries affecting

the upper limbs. These tests are carried out thanks

to the joint development with the Hospital Nacional

de Parapl

ejicos de Toledo in Spain. The study de-

scribed in this article stands out from others because

the comparison between the different environments

(real, non-immersive and immersive) is performed

with the same parameters and sample. In both virtual

environments, the interaction is executed with one’s

own hands, ruling out possible biases introduced by

the use of controllers or other devices. In addition, al-

though the objective of the test is to count the number

of blocks moved correctly, certain variables of inter-

est have been included for further study. On the other

hand, unlike other implementations, immersive VR-

BBT is performed using hand tracking provided by

the virtual reality device itself, thus achieving a more

realistic immersion.

The rest of the article is structured as follows. Sec-

tion 2 sets the context of the problem. Section 3

presents different scenarios and immersive and non-

immersive methods employed to support upper limb

rehabilitation. Section 4 describes the clinical study

and the results obtained. Finally, the article ends with

Section 5 where the conclusions are presented.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

232

2 PROBLEM CONTEXT

2.1 Upper Limb Rehabilitation and

Occupational Therapy

Occupational therapy plays a signiﬁcant role in the re-

habilitation of patients with spinal cord injuries. Ther-

apists assess the most appropriate treatment according

to each individual’s needs and functional goals. The

therapy aims to achieve as much independence as pos-

sible for the patient in daily living activities like eat-

ing, turning lights on and off, dressing, or writing.

The recovery process of patients with SCI is

linked to early mobilization and a multidisciplinary

approach. To achieve improved and optimized long-

term outcomes, it is necessary to involve experts in

nutrition and physical and occupational therapists.

Hence the importance of having an interdisciplinary

team from the beginning of rehabilitation in the hos-

pital.

2.2 Upper Limb Exercises

One of the types of exercises used in upper limb re-

habilitation focuses on improving manipulative skills.

The aim is to achieve the highest possible degree of

hand functionality. In its execution, the hand(s) to be

rehabilitated must be able to grasp and manipulate ob-

jects of different sizes.

Based on the above, there are different classiﬁca-

tions of the movements and grasps required for the

hand to be functional. One such grasp is described by

(Vergara et al., 2014). The classiﬁcation is based on

the common grasps used by adults during the perfor-

mance of activities of daily living: cylindrical grasp

(the palm is involved and the thumb is in abduc-

tion or neutral), oblique palmar grasp (the thumb is

adducted), hook grasp (palm and thumb are not in-

volved), lumbrical grasp (thumb and proximal part

of the ﬁngers), intermediate power-precision grasp,

pinch (thumb and ﬁngertips), lateral pinch (lateral

part of ﬁngers and usually the thumb), special pinch

and non-prehensile grasp (without grasping).

Another type of grasp that is widely used in pa-

tients with tetraplegia is the so-called tenodesis grasp.

This is made by actively extending the wrist, and then

the passive tension of the ﬂexor muscles puts them

under tension, generating a grip between the thumb

and ﬁngers (Jung et al., 2018).

Patients with SCI often need external aids to hold

certain objects, such as a toothbrush or cutlery. In

addition, the supervision by a specialist is essential

when performing the exercises. Incorrect execution

can lead to non-optimal rehabilitation and even to

other secondary problems. These characteristics and

limitations must be taken into account in the develop-

ment of VR applications for rehabilitation.

3 REHABILITATION THROUGH

VIRTUAL REALITY THERAPY

Virtual reality is an emerging tool that is becom-

ing increasingly widespread in the ﬁeld of rehabil-

itation. Studies corroborate the positive effects of

this technology in patients with different types of dis-

eases. Due to the diversity of existing pathologies

and the goals to be achieved, it is difﬁcult to unify

the beneﬁts of virtual reality rehabilitation, although

it is worth highlighting the following: improvement

in motor functions and quality of life (Toldo et al.,

2021), cognitive functions (Maggio et al., 2018), mus-

cle strength (Lee et al., 2016) and increased motiva-

tion (Dias et al., 2019).

Depending on the part of the body to be reha-

bilitated, the devices and virtual environments must

comply with some characteristics. In the case of up-

per limb rehabilitation, it is essential that the user can

move his/her hands freely. Virtual applications that

need controllers to interact with virtual elements do

not simulate natural grasping. Thus, it is necessary to

rely on an efﬁcient hand tracking system from which

it is possible to obtain:

• The position of the hand and ﬁngers in 3D space

with the least possible error.

• The interaction of the hand on the virtual objects

in the environment.

• An objective measurement of the degree of mobil-

ity and achievements of the patient.

In order to meet these needs, the following issues

must be resolved: the ﬁrst is the modeling of the dif-

ferent grasp types and their recognition by the system.

The second is the grab, displacement and release of

the virtual objects. And last, the objective measure-

ment of the results using a standardized test such as

the BBT.

3.1 Modeling Grasp Types

As described in the previous section, various func-

tional grasp allow humans to grasp and move virtual

objects. In the case of virtual reality, this problem is

solved by using a virtual 3D hand that internally relies

on an associated bones and joints (see Figure 1), sim-

ilar to that of a human. In this way, the type of grasp

can be recognized according to the direction vector of

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios

233

Figure 1: Virtual hand in immersive environment with as-

sociated skeleton for accurate ﬁnger position detection.

the bones, position and rotation of the individual ﬁn-

gers in relation to each other or in relation to the palm

of the hand.

In addition to pose recognition, it is also impor-

tant to detect the level of effort. The effort of each

patient can be key in determining when an object is

considered to be attached to a virtual hand. Effort can

be deﬁned as the distance between two or more bones

involved in the grasp. Above a certain threshold, con-

tact is considered to exist between two or more ﬁngers

or between the ﬁngers and palm. On the other hand,

below a certain threshold, it is considered that they is

no contact at all. In the case of grasping a given ob-

ject, it is necessary to take into account both the con-

tact distance between ﬁngers (or ﬁngers and palm of

the hand) and the distance between the hand and the

object. If a grasp pose is recognized but the object is

not within reach, then the grasp will not be executed.

In the case of non-prehensil movements used to

lift an object, it is necessary to know the orientation

of the hand, as it must be placed with the palm facing

the ground.

3.2 Holding of Virtual Objects

In order to simulate, in a virtual environment, the hu-

man ability to grasp and release objects, it is neces-

sary to evaluate the shape and position of the object

in relation to the hand with which it interacts within

the 3D environment.

First of all, the interactive elements of the scene

must be deﬁned along with the kind of interaction.

For this, the physical properties of the object, its

weight or gravity, the collision area of the object and

the state of the object from the interaction point of

view must be speciﬁed. The states related to the inter-

action are as follows: non-selectable (it is away from

the hand), selectable (the hand can interact with it),

grabbed (it is attached to the hand and follows the

movements of the hand) and released (it will be sub-

ject once more to the established physical principles).

In order to recognize the different types of grasps with

respect to the object, the collision area of the object

has to be considered. This area can be the same size

as the object or vary slightly whether a more ﬂuid in-

teraction is required.

Once the object is considered to be grabbed, it is

dependent on the actions of the hand. When grasp

strength is detected to be less than a certain threshold,

the item will behave according to its physical charac-

teristics.

3.3 Box and Block Test (BBT)

Once the different types of grasping and the interac-

tion of the hands with objects in the virtual world have

been established, it is necessary to determine how the

manipulative ability of a user can be measured. There

are different scales and tools to measure motor dex-

terity, such as the Gross Motor test, the Movement

Assessment Battery, the Action Research Arm Test

(ARAT), or the Motor Assessment Scale (MAS) (Carr

et al., 1985).

Particularly, in order to assess manipulative dex-

terity, BBT is widely used by patients with differ-

ent diagnoses: spinal cord injury, geriatrics, multiple

sclerosis and ﬁbromyalgia, among others. Therefore,

in this work, BBT has been performed in three differ-

ent scenarios: a) real physical environment without

the use of technologies, b) non-immersive environ-

ment with virtual elements and, ﬁnally, c) immersive

environment based on virtual reality. In scenarios b)

and c), technological solutions have been adopted so

that the user does not need to have their hands occu-

pied with a controller device or attach wearables to

the body.

3.3.1 Scenario 1: BBT in Physical Environment

The BBT, requires a 53.7 cm x 25.4 cm x 8.5 cm

wooden box divided into two compartments of equal

size. Initially, 150 blue, yellow, or red wooden blocks

with a dimension of 2.5 cm are placed in one of the

compartments. This conﬁguration depends on the

hand to be assessed: if it is the right hand, the blocks

are initially placed in the right compartment; if the

dexterity of the left hand is to be measured, the blocks

are placed in the left compartment.

The patient is seated in front of the box and has to

move as many blocks as possible from one compart-

ment to another within 60 seconds (Mathiowetz et al.,

1985). In order to perform the test, the patient is ﬁrst

given 15 seconds to trial period. Once the test has

started, the examiner must count the displaced blocks

correctly: only one block can be carried in each move-

ment (otherwise, only one is counted), the patient’s

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

234

Figure 2: Analysis of hand movement accuracy and performance in the execution of therapeutic exercises using the Box and

Block test in three types of scenarios: a) physical scenario without the use of technologies, b) use of virtual components in a

non-immersive scenario and, ﬁnally, c) use of virtual components in a fully immersive scenario.

hand must pass over the partition of the box and re-

lease the block when it is in the target partition.

BBT is a simple test, portable, efﬁcient, and in-

expensive to perform. It provides information on the

speed of performance but not on other aspects such as

the quality of the performed movement (Alt Murphy

et al., 2015). There is no single trajectory to perform

the exercise, apart from the different initial positions

of the cubes in the box. In recent years, new tech-

nologies have been adopted to mitigate the disadvan-

tages derived from the test, so that objective measure-

ments of the user’s movements can be obtained. One

of them is the modiﬁed BBT which allows the assess-

ment of movement in upper-limb impairment with the

help of motion capture techniques, and may help mea-

sure the effect of interventions to improve upper-limb

function (Hebert et al., 2014). Other studies make

use of wearable sensors to obtain, among other things,

information on the speed or time spent moving each

block (Zhang et al., 2019).

3.3.2 BBT with Virtual Reality (Scenarios 1 and

Recently, and thanks to the rise of virtual reality, ap-

plications have begun to be developed to simulate

BBT to exploit the advantages of the virtual world

with an accurate capture of movements.

In tests conducted by Gieser et al. (Gieser et al.,

2016) the number of blocks moved in the real BBT

was much higher than those achieved with the non-

immersive VR-BBT. Hashim et al. (Hashim et al.,

2021) in their study with an immersive VR-BBT sup-

ports this result, although this number of achieve-

ments increases in line with the number of training

sessions. Other studies, such as those conducted by

Onat et al. (O

na et al., 5 13) have shown that these

differences are reduced when the environment is im-

mersive, and no additional controllers are used.

This paper presents two solutions to VR-BBT:

non-immersive and immersive. The implementation

of the virtual world has been developed following the

guidelines of the clinical experts of the Hospital de

Parapl

ejicos de Toledo. To this end, the physical char-

acteristics of both the box and the blocks, as well as

the colors used, have been taken into account to elim-

inate possible biases in the results obtained with tra-

ditional BBT and virtual BBT.

Both solutions were developed in Unity

. The

wooden box and blocks were designed following the

guidelines of the real test. A virtual object is modeled

and instantiated as many times as necessary at run-

time to represent the cubes. Each cube has two basic

elements: Rigidbody and Box Collider. The ﬁrst one

allows the cube to act under the control of physics,

deﬁning mass and gravity, among others. The sec-

ond is a box collider that simulates physical collisions

with other elements. In order to be faithful to the real

model, the color of each cube is determined randomly,

according to the colors used in the BBT test: red, blue,

or yellow. The position in the scene is also set ran-

domly, and inside the zone of origin (right side for the

right hand or left side for the left hand).

In both proposals, audio and graphical aids have

https://unity.com/

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios

235

Sensor

Layer

Leap Motion Controller (LMC) Oculus Quest 2

Non-immersive

Immersive VR

Hand

Analysis

Hand Pose

Recognition

Hand Tracking

Hand Pose

Recognition

Hand Tracking

Logic

Layer

BBT Virtual Application BBT Virtual Application

Event Controller

Cube Generator

Physics Engine

Object Gripping Module

Event Controller

Cube Generator

Physics Engine

Object Gripping Module

Visualization

Layer

Screen Display

Visualization

Figure 3: Multilayer architecture of the two proposed solutions for virtual BBT. Main hardware components, software and

connection between modules.

been added to help the patient to know the state of

the interaction. Both immersive and non-immersive

solutions have a sound when the cube is grabbed and

another one when the cube is released. In addition,

when a cube is grabbed then it changes to a greyish

colour.

Moreover, both virtual solutions have a pre-

calibration phase to adjust the object grip. In the event

that a user/patient has severe movement limitations,

the test is run in “autogrip” mode. In other words,

the cube is automatically grasped by proximity to the

hand and will be released when it comes into contact

with the target compartment.

Each of the two VR-based solutions mentioned

above is described in more detail in the following sec-

tion. Figure 3 shows a multilayer architecture of the

two solutions developed. On the left a VR-based and

non-immersive solution, on the right the solution is

fully immersive. The design is divided into four main

layers. The lowest level layer captures the movement

of the hands through a set of sensors. This informa-

tion is processed at a second level to track hands and

recognize grasping gestures. The information gener-

ated at the second level progresses to the logic layer,

where the most signiﬁcant processing load is located.

In the logic layer, there is a module for generating

BBT cubes, the module that determines the grip be-

tween hands and virtual objects, the event controller

module (any given situation in the virtual environ-

ment), and the physics engine to provide the virtual

elements with realistic behavior. Once all the infor-

mation has been processed, the last layer is responsi-

ble for displaying the changes in the virtual environ-

ment. The major difference between the two solutions

is that in the non-immersive solution the changes are

displayed on a screen and the user does not stop per-

ceiving the real world at any time. In contrast, the

second solution feeds back to the ﬁrst layer, as the

changes are displayed again in the VR goggles and

the user is completely isolated from the real world.

3.3.3 Scenario 2: BBT in a Non-Immersive

Scenario With Virtual Reality

In a non-immersive scenario, the patient interacts

with his/her own hands and the results of the ac-

tions are displayed in real-time on a screen. In the

non-immersive virtual BBT presented in this paper,

Leap Moption Controller (LMC) has been used for

hand motion capture (see Figure 4). The Leap Mo-

tion Controller is an optical hand-tracking module

to capture hand and ﬁnger movement. It is a small

size and low-cost sensor with two cameras and in-

frared LEDs. The connection is made via USB 2.0,

it has two 640x240-pixel near-infrared cameras with

infrared-transparent typically operating at 120Hz, but

the hardware is capable of 240+ (lea, 2022). LMC can

track both hands simultaneously, recognizing gestures

and allowing real-time interaction: pushing, pinching,

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

236

grabbing, etc. Besides, Ultraleap offers an API for

different programming languages and supports inte-

gration with both Unity and Unreal engines for build-

ing real-time 3D projects.

Figure 4: BBT in a non-immersive scenario using LMC and

on-screen display.

The different functional hand grasps are detected

when they come into contact with the collision area

deﬁned in the cube. For this implementation (see Fig-

ure 4), the methods provided by the API were used

to determine the grab strength and pinch strength.

Firstly, we calculate the strength of a hand grasping

posture based on how close the hand is to being a

ﬁst. Secondly, the pinch strength indicates the holding

strength of a pinch hand (between the thumb and an-

other ﬁnger of the same hand). In both cases, the do-

main of deﬁnition of the variable is a numerical value

belonging to [0,1]. The value 0 indicates no grip,

whereas the value 1 refers to the highest strength.

A set of variables that inform about the execution

of the exercise is stored in real-time. These variables

are: hand used, reliability (degree of conﬁdence of

the hand), palm position (X,Y,Z), palm position in

magnitude, hand speed, grasp capacity understood

as opening and closing the hand (grasp strength) [0-

1], thumb grip capacity with any of the other ﬁn-

gers (grip strength) [0-1], wrist ﬂexion expressed in

degrees (arm.Direction.AngleTo(hand.Direction) ∗

180/( f loat)π) and the pronation of the forearm from

the hand’s roll angle(normal.Roll ∗ 180/( f loat)π +

90).

Finally, it is worth mentioning that the application

has an initial menu to select the difﬁculty level ac-

cording to time, size, and the number of targets de-

pending on the pinch or grip performed. The menu

design contemplates accessibility aspects so that pa-

tients with physical limitations can interact without

external assistance.

3.3.4 Scenario 3: BBT in Immersive

Environment with Virtual Reality

In immersive VR-BBT, the patient is completely im-

mersed in the virtual world (see Figure 5). The envi-

ronment that the user views and feeds back to is iso-

lated from the real world. Moreover, unlike other pro-

Figure 5: BBT in an immersive scenario in which the pa-

tient perceives only virtual elements.

posals, the solution presented here uses the patient’s

hands as a natural form of interaction. It should be

noted that hand tracking is performed by the virtual

reality device itself, Oculus Quest 2. Unlike other

developments in which was used with virtual reality

glasses (O

na et al., 5 13), the immersive VR-BBT we

propose uses the device’s own hand tracking. From

the clinical study carried out in this work, it was found

that Oculus Quest 2 offer better tracking accuracy

compared to the LMC. This fact results in improved

patient performance as will be discussed in future sec-

tions.

Oculus Quest 2 is a Head-Mounted Display

(HMD) with 6 DoF inside-out via four integrated

cameras for tracking and allows interaction with the

immersive world through the dynamic controllers and

the user’s hands. Hand tracking has been signiﬁcantly

improved in the new version 2.0 (April 2022), which

allows faster movements and solves occlusion issues,

so more applications integrating this type of interac-

tion are starting to emerge. Meta

provides develop-

ers with SDKs for integration into both Unity and Un-

real engines.

The VR-BBT immersive has been developed in

Unity (v.2020.3.23f1) and uses the Hand Tracking

API v2.0. The SDK allows the conﬁguration of cer-

tain parameters; one of the most relevant is the track-

ing rate that can be low, medium or high. The latter

reaches 60 Hz and is the one used in our VR-BBT

application.

The development of the virtual environment has

been implemented following feedback from clinical

specialists. Special attention has been paid to provide

the VR-BBT with a high conﬁgurability so that a wide

variety of patients can use it. The primary consider-

ations taken into account to adapt to each patient’s

needs are listed below.

• Recognition of any possible grasp, including ten-

odesis grasp and release. The block grasp is per-

formed by proximity to the hand in case no func-

https://www.meta.com/

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios

237

tional grasp detected.

• Initial conﬁguration of the positioning of the box.

This static element can be adapted to each patient,

i.e., its position in the three axes (X, Y, Z) can be

modiﬁed before starting the BBT. The box con-

taining the blocks must be positioned at a suitable

height, i.e., the patient must remain seated, with

the box in front of him/her and a 90° angle must be

formed between the arm and forearm. This avoids

forced shoulder, arm, and forearm positions and

even prevents some of the blocks from being in-

accessible. For this reason, the location of the box

can be conﬁgured from the menu, adapting it to

each patient.

• Conﬁguration for patients with severely reduced

mobility. The calibration stage, in which the pa-

tient tries to grasp a sample block in any possible

way might be advisable. The application runs in

“autogrip” mode if no grasp is detected. Concern-

ing the difﬁculty of the game, two forms, “nor-

mal” and “easy”’, have been created, which are

linked to the separator between the two areas of

the box. Some patients cannot perform arm lifts,

but with this conﬁguration it is possible, in easy

mode, to correctly score when the hand crosses

the separator between the two cubicles. These op-

tions allow testing with a wider range of patients.

• Visual and audio feedback: in addition to the el-

ements discussed in section 3.3.2 the immersive

environment has the following elements to sup-

port interaction. The ﬁngertips change to purple

when starting to grasp a block and to green when

it is grabbed by the hand. In addition, the correct

movement of a block is accompanied by a sound.

A different sound is played when the block has

been moved incorrectly or placed outside the tar-

get partition.

• The initial menu is easily accessible. The buttons

are large and can be pressed with the whole hand.

Interaction can be done either with the hands or

with the controllers.

During the execution of the VR-BBT a dataset is

stored to facilitate a possible a posteriori evaluation of

the performed exercise: the number of frames since

the beginning of the exercise, time measured in sec-

onds, hand detected/not detected, the degree of con-

ﬁdence and position in 3D space of the hand, pinch

detection (true/false), palm grip detection (true/false),

force realized the thumb with the rest of the ﬁngers

([0-1]), the strength of each ﬁnger with respect to the

palm ([0-1]) and HMD position (x,y,z). With these

elements, it is possible to determine the number of

correctly performed grips, the type, and the exerted

force. Furthermore, it is possible to detect false neg-

atives due to inefﬁcient hand tracking thanks to the

degree of hand conﬁdence. One of the variables de-

scribed above is worth mentioning, the HMD posi-

tion. This value is stored to report undesirable trunk

compensatory movements. This is done thanks to the

positioning of the HMD in 3D space. The patient’s

head and trunk performs a displacement when move-

ment is detected in the Z axis of the HMD.

The feeling of complete immersion, the adjust-

ments in the virtual environment according to the pa-

tient’s needs and the variables stored at run-time pro-

vide a powerful tool for both the patient and the reha-

bilitators and therapists. The patient feels more mo-

tivated to perform the exercise, which is also adapted

to his or her particular needs. Rehabilitation special-

ists have objective data on the performance of BBT

and can evaluate not only the blocks moved but also

the quality of each movement as well as the patient’s

evolution.

4 CLINICAL STUDY AND

RESULTS

4.1 Participants

The technologies discussed throughout the article

were initially tested on healthy patients. In later

phases of the project, real patients will be included

in the clinical study. Thus, ten healthy individuals

(32.50±17.25 years) participated in the study. All

of them were right-handed and performed the BBT

task in three different experimental conditions with

the dominant hand:

• The real environment.

• The virtual environment in a non-immersive con-

dition by means of Leap Motion Controller.

• The virtual environment in an immersive condi-

tion by means of Oculus Quest 2

With the aim of controlling the order effects, indi-

viduals’ performance in each condition was random-

ized. The clinical study was carried out at the Hospi-

tal Nacional de Parapl

ejicos (Toledo, Spain) and was

approved by the local Ethical Committee.

All the subjects must have neurologically healthy

condition. Exclusion criteria were: not signing the

corresponding informed consent; having visual im-

pairment or any impairment of upper limb function;

having previous history of seizure or motion sickness.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

238

4.2 Clinical Study Setup

Each participant performed all three experimental

conditions on the same day in a single experimental

session. The BBT in each experimental condition was

performed seated in front of a height-adjustable table

until the elbow was ﬂexed 90º with the palm of the

hand on the table. For the immersive VR condition,

the table was removed and it was performed in the

same chair as the other conditions.

Before performing the ﬁrst trial within the both

VR conditions, a preliminary trial was performed to

familiarize with each virtual environment. In the case

of the real BBT, no familiarization was considered

needed with the exception of the 15 seconds allowed

by the real test (Mathiowetz et al., 1985).

Three trials of each experimental condition were

performed. The variable measured was the total num-

ber of cubes passed to the other side of the box in one

minute. As the ﬁnal result, the mean value of the three

trials was considered for analysis for each condition.

4.3 Statistical Analysis and Results

The results of the variable analyzed were expressed as

median and interquartile range. The Wilcoxon non-

parametric test was applied to ﬁnd possible differ-

ences between the three experimental conditions (see

table 1). The relation between each pair of conditions

was analyzed by the Pearson correlation coefﬁcient.

The performance in the real BBT was signiﬁcantly

higher than in the both VR modalities (83.67 ± 16.75

in the real test vs. 69.17 ± 20.33 in the immersive

VR version (p < 0.05) and 45.66 ± 16.84 in the non-

immersive version (p < 0.01)) (see Figure 8). More-

over these results were statistically signiﬁcant be-

tween the both VR modalities (p < 0.01) (see Fig-

ure 7). However, if we selected only participants with

previous experience in virtual environments (n = 8),

no statistically signiﬁcant differences were found be-

tween the real BBT and the immersive VR version

(see Figure 6).

The correlation between the real BBT and the non-

immersive VR BBT was high (0.858, p < 0.05). This

correlation was statistically signiﬁcant (p < 0.01) be-

tween the real BBT and the immersive BBT (0.717)

and between both VR conditions (0.763).

4.4 Limitations of the Study

The conducted clinical study has been oriented to-

wards the evaluation of data acquisition and perfor-

mance in relation to the BBT. This is an essential as-

pect when devising a system that allows autonomous

Figure 6: Box plot in relation to the cubes passed for each

experimental condition with practice in VR environments

(n = 8).

Figure 7: Box plot in relation to the cubes passed for each

experimental condition and for all the participants (n = 10).

but guided rehabilitation. This ﬁrst step is key before

evaluating its usefulness with real patients who have

suffered neurological conditions.

After this ﬁrst step, a series of clinical studies to

analyze if this trend is maintained in spinal cord in-

jured patients will be conducted in a second phase.

These are intended to be performed in the Hospital

Nacional de Parapl

ejicos, where the system will be

deployed.

The technological requirements of the discussed

system are not high (VR headset and an standard lap-

top). The authors of this manuscript do not consider

the system as high-cost, since it aims at improving

the rehabilitation process and mitigating the lack of

specialized staff, which is becoming more and more

challenging from a global point of view (see (Alvarez-

Figure 8: Performance of each participant within each ex-

perimental condition. All the participants maintain the same

trend.

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios

239

Table 1: Signiﬁcant statistically differences between the three experimental environments analyzed by means of Wilcoxon

test (a,b(p < 0.05), c,d(p < 0.01)).

Participants Physical environment Non-immersive VR Immersive VR

All (n=10) 83.67(16.75)

a,c

45.66(16.84)

c,d

69.17(20.33)

a,d

With practice (n= 8) 83.67(13.50)

45.66(15.50)

a,b

72.17(21.83)

Sab

ın et al., 2017) for a related discussion that affects

stroke rehabilitation in Spain). At this point, one of

the pursued goals is to show the usefulness, and even

the acceptance, of the system when performing reha-

bilitation. Particularly, the study and analysis the in-

ﬂuence of motivation, in the short term, on the user’s

commitment has been considered.

Lastly, in a third phase, a clinical trial will be

conducted to study the feasibility and effectiveness

of the proposed system, considering its intended use

during longer periods of time. The research hypoth-

esis at this point would be the assessment and vali-

dation of effective, upper-limb rehabilitation through

virtual reality at hospitals and rehabilitation centers.

In this phase, functional resonance magnetic imag-

ing (fRMI) will be included in the study for detect-

ing signs of improvement after the rehabilitation treat-

ment. This feature will allow us to design and de-

scribe a motor paradigm and the corresponding ex-

perimental protocol, suitable for all the participants,

whether healthy or SCI patients. Finally, to extend the

fMRI post-processing for analyzing voxel-based mor-

phometry and voxel-based thickness of T1-weighted

images.

5 CONCLUSIONS

This paper analyzes and compares technologies that

facilitate hand tracking and are applied to upper limb

rehabilitation. For this purpose, non-immersive and

immersive virtual solutions based on virtual reality

have been developed and applied in the context of the

Hospital Nacional de Parapl

ejicos de Toledo (Spain).

The proposed solutions are integrated together

with the BBT and three different scenarios are pro-

posed: traditional physical scenario without the use

of technologies, non-immersive scenario and virtual

elements and, ﬁnally, fully immersive scenario based

on VR.

The main novelty of the work lies in the simulta-

neous design of both non-immersive and immersive

solutions that take into account the future limitations

of patients. Precisely because of these limitations, we

have opted for implementations in which patients can

perform the exercises without the attachment of real

physical controllers or the adhesion of wearables de-

vices.

The starting hypothesis was that the conditions of-

fered by immersive virtual environments can recreate

scenarios and situations very similar to real ones, to

which should be added the advantages of these virtual

environments such as the elimination of physical bar-

riers and the recreation of any type of object. In the

context of rehabilitation, this factor is crucial since

the precision with which therapeutic exercises are per-

formed directly inﬂuences the recovery of damaged

limbs.

Precisely, the results demonstrate this hypothesis.

The number of blocks moved from the point of origin

to the point of destination in the immersive environ-

ment largely resembles the data obtained in a tradi-

tional physical environment. This is due to a proper

perception of the environment and the elements pre-

sented, the correct implementation of grasping pose

recognition and, ﬁnally, the control logic that deter-

mines the adhesion of objects to the hands.

As future work, the use of devices that can mea-

sure the muscular activity exerted by the patient dur-

ing virtual rehabilitation is proposed, using, for ex-

ample, EMG-based controllers (Pleva et al., 2022).

As well as mixed reality technologies that would al-

low for greater precision in measuring the patient’s

progress and better adaptation of the intensity and dif-

ﬁculty of the exercises to the individual needs of the

patient.

ACKNOWLEDGMENTS

This work has been founded by the Spanish Ministry

of Science, Innovation and Universities under the Re-

search Project: Platform for Upper Extremity Reha-

bilitation based on Immersive Virtual Reality (Rehab-

Immersive), PID2020-117361RB-C21 and PID2020-

117361RB-C22.

REFERENCES

(2022). Leap Motion Controller Datasheet. Ultraleap.

Alt Murphy, M., Resteghini, C., Feys, P., and Lamers, I.

(2015). An overview of systematic reviews on upper

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

240

extremity outcome measures after stroke. BMC Neu-

rology, 15(1):29.

Alvarez-Sab

ın, J., Quintana, M., Masjuan, J., Oliva-

Moreno, J., Mar, J., Gonzalez-Rojas, N., Becerra, V.,

Torres, C., and Yebenes, M. (2017). Economic impact

of patients admitted to stroke units in spain. The Eu-

ropean Journal of Health Economics, 18(4):449–458.

Bravo, V. P. and Mu

noz, J. A. (2022). Wearables and their

applications for the rehabilitation of elderly people.

Medical and Biological Engineering and Computing,

60(5):1239—1252.

Carr, J. H., Shepherd, R. B., Nordholm, L., and Lynne, D.

(1985). Investigation of a new motor assessment scale

for stroke patients. Physical Therapy, 65(2):175–180.

Clus, D., Larsen, M. E., Lemey, C., and Berrouiguet,

S. (2018). The Use of Virtual Reality in Patients

with Eating Disorders: Systematic Review. Journal

of Medical Internet Research, 20(4):e7898. Com-

pany: Journal of Medical Internet Research Distribu-

tor: Journal of Medical Internet Research Institution:

Journal of Medical Internet Research Label: Journal

of Medical Internet Research Publisher: JMIR Publi-

cations Inc., Toronto, Canada.

de Ara

ujo, A. V. L., Neiva, J. F. d. O., Monteiro, C. B.

d. M., and Magalh

aes, F. H. (2019). Efﬁcacy of Vir-

tual Reality Rehabilitation after Spinal Cord Injury: A

Systematic Review. BioMed Research International,

2019:e7106951. Publisher: Hindawi.

Dias, P., Silva, R., Amorim, P., La

ıns, J., Roque, E.,

Ser

odio, I., Pereira, F., and Santos, B. S. (2019). Using

Virtual Reality to Increase Motivation in Poststroke

Rehabilitation. IEEE Computer Graphics and Appli-

cations, 39(1):64–70. Conference Name: IEEE Com-

puter Graphics and Applications.

Everard, G., Otmane-Tolba, Y., Rosselli, Z., Pellissier, T.,

Ajana, K., Dehem, S., Auvinet, E., Edwards, M. G.,

Lebleu, J., and Lejeune, T. (2022). Concurrent valid-

ity of an immersive virtual reality version of the box

and block test to assess manual dexterity among pa-

tients with stroke. Journal of NeuroEngineering and

Rehabilitation, 19(1):7.

Gieser, S. N., Gentry, C., LePage, J., and Makedon, F.

(2016). Comparing objective and subjective metrics

between physical and virtual tasks. In Lackey, S. and

Shumaker, R., editors, Virtual, Augmented and Mixed

Reality, Lecture Notes in Computer Science, pages 3–

13. Springer International Publishing.

Guill

en-Climent, S., Garzo, A., Mu

noz Alcaraz, M. N.,

Casado-Adam, P., Arcas-Ruiz-Ruano, J., Mej

ıas-

Ruiz, M., and Mayordomo-Riera, F. J. (2021). A us-

ability study in patients with stroke using MERLIN, a

robotic system based on serious games for upper limb

rehabilitation in the home setting. Journal of Neuro-

Engineering and Rehabilitation, 18(1):41.

Hashim, N. A., Razak, N. A. A., and Osman, N. A. A.

(2021). Comparison of Conventional and Virtual Re-

ality Box and Blocks Tests in Upper Limb Amputees:

A Case-Control Study. IEEE Access, 9:76983–76990.

Conference Name: IEEE Access.

Hebert, J. S., Lewicke, J., Williams, T. R., and Vette, A. H.

(2014). Normative data for modiﬁed box and blocks

test measuring upper-limb function via motion cap-

ture. Journal of Rehabilitation Research and Devel-

opment, 51(6):918–932.

Hodkin, E. F., Lei, Y., Humby, J., Glover, I. S., Choud-

hury, S., Kumar, H., Perez, M. A., Rodgers, H., and

Jackson, A. (2018). Automated fes for upper limb

rehabilitation following stroke and spinal cord injury.

IEEE Transactions on Neural Systems and Rehabili-

tation Engineering, 26(5):1067–1074.

Jung, H. Y., Lee, J., and Shin, H. I. (2018). The nat-

ural course of passive tenodesis grip in individuals

with spinal cord injury with preserved wrist extension

power but paralyzed ﬁngers and thumbs. Spinal Cord,

56(9):900–906.

Kang, Y., Ding, H., Zhou, H., Wei, Z., Liu, L., Pan, D.,

and Feng, S. (2017). Epidemiology of worldwide

spinal cord injury: a literature review. Journal of Neu-

rorestoratology, Volume 6:1–9.

Lamash, L., Klinger, E., and Josman, N. (2017). Using a

virtual supermarket to promote independent function-

ing among adolescents with Autism Spectrum Disor-

der. In 2017 International Conference on Virtual Re-

habilitation (ICVR), pages 1–7. ISSN: 2331-9569.

Lee, S.-H., Yeh, S.-C., Chan, R.-C., Chen, S., Yang, G.,

and Zheng, L.-R. (2016). Motor ingredients derived

from a wearable sensor-based virtual reality system

for frozen shoulder rehabilitation. BioMed Research

International, 10(6). Publisher: Hindawi.

Maggio, M. G., De Cola, M. C., Latella, D., Maresca, G.,

Finocchiaro, C., La Rosa, G., Cimino, V., Sorbera, C.,

Bramanti, P., De Luca, R., and Calabr

o, R. S. (2018).

What about the role of virtual reality in parkinson dis-

ease’s cognitive rehabilitation? preliminary ﬁndings

from a randomized clinical trial. Journal of Geriatric

Psychiatry and Neurology, 31(6):312–318. Publisher:

SAGE Publications Inc STM.

Mathiowetz, V., Volland, G., Kashman, N., and Weber, K.

(1985). Adult norms for the box and block test of man-

ual dexterity. The American Journal of Occupational

Therapy, 39(6):386–391.

Mekki, M., Delgado, A. D., Fry, A., Putrino, D., and

Huang, V. (2018). Robotic rehabilitation and spinal

cord injury: a narrative review. Neurotherapeutics,

15(3):604–617.

na, E. D., Jard

on Huete, A., Cuesta-G

omez, A., Baeza,

P., Cano de la Cuerda, R., and Balaguer, C. (2020-05-

13). Validity of a fully-immersive VR-based version

of the box and blocks test for upper limb function as-

sessment in parkinson’s disease. Sensors, 20:2773.

Oliveira, C. S., Almeida, C. S., Freitas, L. C., Santana, R.,

Fernandes, G., Junior, P. R. F., and Moura, R. C. F.

(2016). Use of the box and block test for the evalu-

ation of manual dexterity in individuals with central

nervous system disorders: A systematic review. Man-

ual Therapy, Posturology & Rehabilitation Jour-

nal, pages 1–7.

Pereira, M. F., Prahm, C., Kolbenschlag, J., Oliveira, E.,

and Rodrigues, N. F. (2020). A virtual reality serious

game for hand rehabilitation therapy. In 2020 IEEE

8th International Conference on Serious Games and

Performance Analysis for Upper Limb Rehabilitation in Non-Immersive and Immersive Scenarios

241

Applications for Health (SeGAH), pages 1–7. ISSN:

2573-3060.

Pleva, M., Koare

cko, v., Hladek, D., Bours, P., Skudal,

M. H., and Liao, Y.-F. (2022). Biometric user iden-

tiﬁcation by forearm EMG analysis. In 2022 IEEE

International Conference on Consumer Electronics -

Taiwan, pages 607–608. ISSN: 2575-8284.

Rupp, R., Biering-Sørensen, F., Burns, S. P., Graves, D. E.,

Guest, J., Jones, L., Read, M. S., Rodriguez, G. M.,

Schuld, C., Tansey-Md, K. E., et al. (2021). Inter-

national standards for neurological classiﬁcation of

spinal cord injury: revised 2019. Topics in spinal cord

injury rehabilitation, 27(2):1–22.

Spooren, A. I. F., Janssen-Potten, Y. J. M., Kerckhofs, E.,

Bongers, H. M. H., and Seelen, H. A. M. (2011). Eval-

uation of a task-oriented client-centered upper extrem-

ity skilled performance training module in persons

with tetraplegia. Spinal Cord, 49(10):1049–1054.

Toldo, J. M. P., Arjona, M., Campos Neto, G. C., Vitor, T.,

Nogueira, S. A., Amaro, E. J., Saba, R. A., Silva, S.

M. C. A., Ferraz, H. B., and Fel

ıcio, A. C. (2021). Vir-

tual rehabilitation in parkinson disease: A dopamine

transporter imaging study. American Journal of Phys-

ical Medicine & Rehabilitation, 100(4):359–366.

Vergara, M., Sancho-Bru, J. L., Gracia-Ib

nez, V., and

erez-Gonz

alez, A. (2014). An introductory study of

common grasps used by adults during performance of

activities of daily living. Journal of Hand Therapy,

27(3):225–234.

Wyndaele, M. and Wyndaele, J.-J. (2006). Incidence, preva-

lence and epidemiology of spinal cord injury: what

learns a worldwide literature survey? Spinal cord,

44(9):523–529.

Zhang, Y., Chen, Y., Yu, H., Lv, Z., Shang, P., Ouyang,

Y., Yang, X., and Lu, W. (2019). Wearable

sensors based automatic box and block test sys-

tem. In 2019 IEEE SmartWorld, Ubiquitous

Intelligence Computing, Advanced and Trusted

Computing, Scalable Computing and Communi-

cations, Cloud and Big Data Computing, Internet

of People and Smart City Innovation (Smart-

World/SCALCOM/UIC/ATC/CBDCom/IOP/SCI),

pages 952–959.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

242