Evaluating Player Performance and Experience in Virtual Reality Game

Interactions using the HTC Vive Controller and Leap Motion Sensor

Diego Navarro and Veronica Sundstedt

Blekinge Institute of Technology, Karlskrona, Sweden

Keywords:

Game Interaction, HTC Vive, Leap Motion, Virtual Reality, User Study, Performance, Experience.

Abstract:

An important aspect of virtual reality (VR) interfaces are novel natural user interactions (NUIs). The increased

use of VR games requires the evaluation of novel interaction techniques that allow efﬁcient manipulations of

3D elements using the hands of the player. Examples of VR devices that support these interactions include the

HTC Vive controller and the Leap Motion sensor. This paper presents a quantitative and qualitative evalua-

tion of player performance and experience in a controlled experiment with 20 volunteering participants. The

experiment evaluated the HTC Vive controller and the Leap Motion sensor when manipulating 3D objects in

two VR games. The ﬁrst game was a Pentomino puzzle and the second game consisted of a ball-throwing

task. Four interaction techniques (picking up, dropping, rotating, and throwing objects) were evaluated as part

of the experiment. The number of user interactions with the Pentomino pieces, the number of ball throws, and

game completion time were metrics used to analyze the player performance. A questionnaire was also used

to evaluate the player experience regarding enjoyment, ease of use, sense of control and user preference. The

overall results show that there was a signiﬁcant decrease in player performance when using the Leap Motion

sensor for the VR game tasks. Participants also reported that hand gestures with the Leap Motion sensor were

not as reliable as the HTC Vive controller. However, the survey showed positive responses when using both

technologies. The paper also offers ideas to keep exploring the capabilities of NUI techniques in the future.

1 INTRODUCTION

Virtual reality (VR) has become a popular consumer

technology thanks to its mass production and constant

development of novel interactive content tailored for

it. VR offers a high level of immersion due, in part,

to its coverage of the full peripheral view of users

and head tracking capabilities. Despite this, the ini-

tial level of immersion experienced with VR tech-

nology can be deteriorated when users fail to inter-

act naturally and effectively with the virtual environ-

ments (Kato et al., 2000; Lee et al., 2017; Bach-

mann et al., 2018). Hardware manufacturers, includ-

ing HTC, have addressed this issue by implement-

ing direct hand manipulation through a set of propri-

etary peripherals, adapting design concepts from tra-

ditional gamepad controllers (buttons, triggers, rum-

ble packs and wireless connectivity) into hand-held

devices suitable for 3D-space interaction. The HTC

Vive controller is one example of them. More recent

designs have tried to further enhance the user experi-

ence by introducing natural user interactions (NUIs),

techniques that recreate real-life action within a cer-

tain level of ﬁdelity (McMahan et al., 2010), as a

novel method for interacting with VR environments.

An example of them is the Leap Motion sensor, which

allows hand tracking and natural hand manipulation

of virtual objects. Different VR interactive applica-

tions have featured novel interaction techniques using

the HTC Vive controller and the Leap Motion sen-

sor. Nonetheless, despite previous contributions, few

studies offer a detailed evaluation of how user per-

formance and experience could be affected by their

use. This situation offers an opportunity to analyze

how efﬁcient NUIs could be when performing basic

interactive tasks in VR environments, in comparison

to hand-held devices. To contribute to the previous

gap, the following study proposes a couple of research

questions: (1) What are the variations in player per-

formance between the HTC Vive controller and the

Leap Motion sensor when used as the main inter-

action input for picking up, dropping, rotating and

throwing objects in VR games? and (2) What are

the variations in the perceived player experience be-

tween the HTC Vive controller and the Leap Motion

sensor when analyzed in terms of enjoyment, ease

Navarro, D. and Sundstedt, V.

Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor.

DOI: 10.5220/0007362401030110

In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), pages 103-110

ISBN: 978-989-758-354-4

103

of use, sense of control and user preference?. An

initial hypothesis proposes that the HTC Vive con-

troller will offer a better overall performance result,

compared with the Leap Motion sensor. However,

there will not be a signiﬁcantly large difference be-

tween the results obtained with both sensors and the

Leap motion will be the preferred one by participants.

The previous questions are addressed by performing

a user study, that differentiates from similar contri-

butions (e.g. (Caggianese et al., 2019)) by offering

a more complex game environment in which track-

ing precision and accurate control over the interaction

techniques play a greater factor in the performance

measurements. Results from this study could help to

identify some of the possibilities and challenges, of-

fered by NUIs and hand-held controllers, in the ﬁeld

of efﬁcient 3D-interaction techniques. Additionally,

it can provide an initial understanding of how these

technologies can affect the overall perceived user ex-

perience when used in VR environments and other vi-

sual and interactive applications.

2 BACKGROUND

The HTC Vive controller is a hand-held motion and

pointing device that resembles a television remote

controller with a hollow ring attached to its top. It

features a track-pad in the front, a couple of buttons

on either side and a trigger button in the back side

of its handle. It uses a set of 24 infra-red (IR) sen-

sors for positional tracking with the HTC Vive base

stations (devices responsible for the tracking of the

headset and controllers), and operates in a frequency

ranging between 250Hz and 1KHz

. On the other

hand, the Leap Motion sensor is a small rectangular

device that combines three sources of IR light, to-

gether with a stereoscopic array of IR cameras, ﬁtted

inside of a metallic frame

. It allows the tracking of

hand movements, with a 5-ﬁnger level of detail, with-

out the need of wearing any additional hardware. It

operates at a frequency of 200 frames/sec, with a ﬁeld

of view of 150

◦

, an interaction area of 0.74m

approx-

imately and an overall accuracy of 0.7mm (Weichert

et al., 2013; Hornsey and Hibbard, 2015). Previous

contributions analyzing VR technologies, in combi-

nation with NUI devices, have focused on exploring

their interactive capabilities in a wide variety of sce-

narios, while also identifying some of their poten-

tial drawbacks and limitations (Peter Wozniak, 2016).

https://www.vrheads.com/exposing-magic-behind-htc-

vive-controller, visited October 2018.

https://learn.sparkfun.com/tutorials/leap-motion-

teardown, visited October 2018.

Nonetheless, only a small set of recent studies offer

a direct comparison between the HTC Vive controller

and the Leap Motion sensor, evaluating quantitative

and qualitative variables from the user performance

and experience, that derives from their respective use.

An initial usability evaluation analyzed two dif-

ferent interaction techniques for manipulating objects

in a VR environment. The techniques focused on

translating and rotating virtual objects through direct

(touching the object) and constrained (using a visual

pivot attached to the object) manipulation. The results

were qualitatively analyzed and showed that partici-

pants preferred to use direct manipulation for trans-

lating objects, while constrained manipulation was

the preferred method for rotation (Caggianese et al.,

2016).

A performance analysis was conducted on a col-

laborative virtual environment. In this study, pulling

and pushing interactions were tested by 30 partici-

pants in an exergame setup. Two participants took the

test at a time. The ﬁrst one was immersed in the vir-

tual environment while the second monitored the per-

formance of the ﬁrst participant and provided visual

assistance to complete the task. Performance results

were analyzed quantitatively and showed an overall

better performance of the HTC Vive controller over

the Leap Motion sensor, requiring a less amount of

interactions and less time to complete the experimen-

tal task (Gusai et al., 2017).

A similar study evaluated the variations in user ex-

perience when interactively manipulating 3D graphs,

comparing traditional input methods (gamepads,

mouse and keyboard) against natural interaction tech-

niques with the Leap Motion sensor. Results have

shown that participants had an overall better experi-

ence when using the traditional input methods, sug-

gesting ease of use and responsive control to be some

of the determining factors for such an outcome. Even

if participants found the use of natural interaction

techniques interesting and fun, the limitations of the

technology and the variations in user preference for

performing and articulating gestures made the Leap

Motion sensor a challenging platform to manipulate

3D graphs (Erra et al., 2018).

Finally, a more recent study analyzed the possi-

bilities each device offered in terms of interaction

design, and evaluated their respective performance

when used for object manipulation in virtual environ-

ments. Results from this study show an overall user

preference for the HTC Vive controller, while identi-

fying a necessity for simplifying complex interaction

techniques in virtual environments, to deliver an over-

all better user experience (Caggianese et al., 2019).

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

104

3 METHODOLOGY

A user study was selected as the main methodology

for the research. Volunteering participants were ex-

posed to two different VR games featuring either the

HTC Vive controller or the Leap Motion sensor. Dur-

ing gameplay, different data was captured as quantita-

tive metrics for evaluating the player performance. A

survey, consisting of a modiﬁed version of the Game

Experience Questionnaire (IJsselsteijn et al., 2013),

was presented to the participants in order to ana-

lyze the perceived player experience upon completing

each game level.

3.1 Participant Criteria

The participant criteria aimed to provide a more ho-

mogeneous group in terms of age, previous experi-

ences with VR and the evaluated game genres. It was:

(1) experience with VR content of at least 2 hours, (2)

experience using the HTC Vive controller of at least 2

hours, (3) experience with puzzle games (e.g. Tetris,

World of Goo, Antichamber or Lyne) of at least 10

hours, (4) an age range between 20 and 28 years old,

(5) no previous cases of photo-sensitivity or epilepsy

and (6) no other medical or physiological condition

that limits the use of stereo vision or the use of hands

and ﬁngers. Only participants that self-reported they

fulﬁlled these criteria were included.

3.2 Experiment Procedure

The experimentation was a within-participant user

study, with one factor and two levels of repeated mea-

sures. Volunteers were initially presented with a con-

sent form, describing the inclusion criteria, goal of

the study, experimental procedure, stimuli, tasks, re-

quired time, risk of participation, data captured and

data protection policies. Upon agreeing to and sign-

ing the form, the experimenter then verbally intro-

duced the two interaction devices used for the test and

guided the participant to the gameplay area (see Sec-

tion 3.3). The HTC Vive headset was placed upon the

participant head and, since the interaction devices are

capable of tracking both hands, participants were told

the study should be performed using only their dom-

inant one. A couple of VR games were presented to

the participants as the main stimuli. One game fea-

tured the HTC Vive controller as the primary inter-

action device, while the other used the Leap Motion

sensor. The VR games, composed by a tutorial, a Pen-

tomino puzzle level and a Ball-throwing level (shown

in Figure 1 and Figure 2), were identical in content,

being the interaction devices they featured, together

with their respective interaction techniques, and the

reference image provided for the Pentomino level the

only variations between them. Both games were de-

veloped using the Unity game engine, v. 2018.2.7f1.

The HTC Vive headset and controller were integrated

using the Steam VR plugin for Unity, v. 2.0.1, by

Valve Corp. For controlling the Leap Motion sensor,

the Unity Assets for Leap Motion Orion Beta v. 4.4.0

and the Leap Motion Interaction Engine Module for

Unity v. 1.2.0, both published by Leap Motion Inc,

were used.

The tutorial level was presented ﬁrst and intro-

duced the participants to the virtual environment, the

3D elements and the four main interaction techniques

evaluated in the study: picking up, rotating, dropping

and throwing virtual objects. Participants were ver-

bally guided through the tutorial level, receiving ex-

planations about the rules of the game, experimental

conditions, tasks and how to perform them correctly

using the respective interaction device. Participants

were allowed to proceed to the following levels only

when they manifested a complete understanding and

sense of control over the game tasks and how to use

the interaction devices. In this way, the lack of expe-

rience with the game or unfamiliarity with the inter-

action techniques were minimized.

The Pentomino puzzle was the ﬁrst game level

participants were exposed to, being tasked with as-

sembling a diamond-shaped ﬁgure by picking, drop-

ping and rotating a set of 12 Pentomino pieces (ge-

ometric shapes composed of ﬁve cubes of the same

size, connected through one or several faces

) as

fast as possible. A reference image was offered as

a visual aid, showing the correct arrangement of the

pieces. The same diamond shape was used in both

games, but the original reference image was ﬂipped

horizontally and vertically in the Leap Motion game

(see inset in Figure 1) to control for carryover effects

without the risk of increasing the difﬁculty of the task.

The Ball-throwing level was shown last, tasking play-

ers with hitting six targets, ﬂoating in the horizon, by

throwing a set of three balls at them. The task had

to be completed in the least amount of time possible.

Targets were placed at different distances and heights

from the initial viewpoint of the participants, while

the balls appeared directly in front of them. Players

were surrounded by leaned walls with no friction, so

balls could rapidly bounce back to the their position.

To grab a piece with the HTC Vive controller, par-

ticipants needed to press and hold the trigger button

while touching its surface with the top ring of the

controller. Releasing the trigger button dropped the

piece. For rotating, participants needed to physically

https://en.wikipedia.org/wiki/Pentomino

Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor

105

Figure 1: The Pentomino puzzle level, using the HTC Vive

controller. Inset in the lower-right corner shows the refer-

ence images used for the Pentomino levels.

Figure 2: The Ball-throwing level, using the Leap Motion

sensor. Inset in the lower-right corner shows the HTC Vive,

its controller and the Leap Motion sensor attached to the

front of the headset.

rotate the controller while holding the piece in their

hands. To perform these actions with the Leap Motion

sensor, participants needed to recreate similar hand

movements as they would usually do in the real world.

To grab a piece, the thumb and any other ﬁnger from

the participants’ hand needed to be in contact with its

surface. Participants were instructed to grab pieces

by pinching or claw-gripping them to avoid tracking

issues with the sensor since a closed ﬁst showed an er-

ratic behavior during initial tests. To release a piece,

participants simply opened their hand widely. To ro-

tate a piece, participants needed to hold it in their hand

while rotating it. For throwing a ball, participants had

to release it while performing the throwing motion.

Upon completing each of the VR games, the head-

set was removed from the participants, and a modiﬁed

version of the Game Experience Questionnaire (IJs-

selsteijn et al., 2013) was presented. The survey

evaluated their perceived experience when interacting

with the virtual world, by performing each of the eval-

uated tasks with the respective interaction device. The

game stimuli were presented counterbalanced to the

participants in an alternated order. This means that

if the current participant started with the game fea-

turing the HTC Vive controller, the next started with

the game using the Leap Motion sensor. When both

games were completed, the ﬁnal part of the survey

was given. The ﬁnal part evaluated their overall opin-

ion and preferences when having a general compari-

son between both devices.

3.3 Experimental Setup and Equipment

The study was conducted within a dedicated con-

trolled environment, provided by the Department of

Creative Technologies (DIKR) at the Blekinge Insti-

tute of Technology. Following the criteria for an ap-

propriate exposure of VR content proposed by (Lopez

et al., 2017), a dedicated squared gameplay area of 2.5

x 2.5 meters, free of any obstacles, was prepared for

the participants. Also, participants were seated during

the entire length of the experimentation.

Outside the gameplay area, a dedicated worksta-

tion computer ran the VR games while, simultane-

ously, managing the different apparatus needed for

the experimentation. The computer had an Intel Core

i7 6700k CPU @ 4.00GHz, a Nvidia GeForce GTX

980 GPU and 16 GB of Corsair DDR4 RAM @

1333MHz. The apparatus used for this study were the

HTC Vive VR headset, its base stations, controller,

and the Leap Motion sensor. The HTC Vive controller

was held in the participants’ hands and was attached

to their wrists. The Leap Motion sensor was attached

to the headset using the Universal VR Dev Mount that

was glued to the front of it (see inset in Figure 2).

4 RESULTS

A total of 20 participants volunteered for the study.

Gathered data were classiﬁed into performance (Sec-

tion 4.1) and experience results (Section 4.2). Sta-

tistical signiﬁcance tests were applied to the gathered

performance data, while the survey results were an-

alyzed according to the scoring parameters exposed

in the Game Experience Questionnaire (IJsselsteijn

et al., 2013).

4.1 Performance Results

The total amount of pieces grabs needed to com-

plete the game, together with the total completion

time, were the evaluation metrics for the performance

analysis in the Pentomino level. The Ball-throwing

level evaluated the total number of throws required

to ﬁnish the level, along with the completion time.

Performance data was initially tested to verify the

ANOVA assumptions and determine an appropriate

signiﬁcance test. Normality was evaluated using

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

106

Pentomino Puzzle Level Results

100

120

140

160

180

200

220

240

260

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Grabs

Participant

(A) Amount of Pieces Grabbed

HTC Vive

Leap Motion

100

120

140

160

HTC Vive Leap Motion

Grabs

Device

(B) Average Grabs

120

180

240

300

360

420

480

540

600

660

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Seconds

Participant

HTC Vive

Leap Motion

120

180

240

300

360

420

HTC Vive Leap Motion

Seconds

Device

(D) Average Completion Time

Figure 3: Performance summary for the Pentomino puzzle level results. Figure (A) shows the total amount of pieces grabs per

participant and device. Figure (B) shows the average number of grabs per device. Figure (C) shows the total level completion

time per participant and device. Figure (D) shows the average completion time per device.

Ball-Throwing Level Results

100

110

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Throws

Participant

(A) Balls Thrown

HTC Vive

Leap Motion

HTC Vive Leap Motion

Throws

Device

(B) Average Throws

120

150

180

210

240

270

300

330

360

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Seconds

Participant

HTC Vive

Leap Motion

120

150

180

HTC Vive Leap Motion

Seconds

Device

(D) Average Completion Time

Figure 4: Performance summary for the Ball-throwing level results. Figure (A) shows the total amount of throws per partici-

pant and device. Figure (B) shows the average number of throws per device. Figure (C) shows the total level completion time

per participant and device. Figure (D) shows the average completion time per device.

the Shapiro-Wilk test on data residuals, while ho-

moscedasticity was reviewed with the Levene’s test.

None of the gathered data was able to satisfy the

ANOVA assumptions. Therefore a non-parametric

test was applied. Based on the data properties and

the proposed experimental design, a Wilcoxon-Mann-

Withney (WMW) test was used for the statistical sig-

niﬁcance evaluation.

4.1.1 Pentomino Puzzle Level

The summary of the results obtained in the Pentomino

puzzle level can be seen in Figure 3. Overall, partic-

ipants had better performance when using the HTC

Vive controller, requiring fewer pieces grabs and time

to complete the level. For the pieces grabs, partici-

pants averaged 50.1 when using the HTC Vive con-

troller, compared to 91.35 with the Leap Motion sen-

sor. After applying the WMW test, results showed

a statistically signiﬁcant difference between the HTC

Vive controller and the Leap Motion sensor.

W MW

(1,n=20)

= 3.7355, p < 0.05

For completion times, participants required more

time to complete the game using the Leap Motion

sensor, scoring an average of 251.48 seconds. The

time needed with the HTC Vive controller was less,

averaging 135.29 seconds. Results from the WMW

test showed a statistically signiﬁcant difference in the

completion times for each interaction device.

W MW

(1,n=20)

= 5.1395, p < 0.05

4.1.2 Ball-throwing Level

A summary of the results gathered from the Ball-

throwing level is illustrated in Figure 4. Participants

had overall better performance when using the HTC

Vive controller, needing fewer throws and less time to

complete the game, when compared with the scores

obtained with the Leap Motion sensor. The average

amount of throws needed to complete the level us-

ing the HTC Vive controller was 10.1, while 30.45

was the average needed with the Leap Motion sensor.

The WMW test showed a statistically signiﬁcant dif-

ference between the number of throws needed with

each interaction device.

W MW

(1,n=20)

= 4.3283, p < 0.05

The time required to ﬁnish the level was higher

when using the Leap Motion sensor, with an average

of 107.73 seconds among the participants, when com-

pared with the average time of 33.61 seconds obtained

with the HTC Vive controller. Time results had a

statistically signiﬁcant difference between them when

evaluated with the WMW test.

W MW

(1,n=20)

= 4.0846, p < 0.05

Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor

107

4.2 Experience Results

The survey was composed by three parts: Part 1 eval-

uated the game featuring the HTC Vive controller,

Part 2 focused on the game using the Leap Motion

sensor and Part 3 made a direct comparison between

both interaction devices. Each part of the survey was,

subsequently, composed of two modules. For Part 1

and 2, there was a module for the Pentomino puzzle

and the ball throwing levels. In Part 3, each interac-

tion device had its module respectively. Each module

from Part 1 and Part 2 presented 11 different state-

ments to participants. They evaluated ﬁve compo-

nents from the perceived experience: Statements 1,

3 and 5 evaluated competence, Statements 2 and 7

evaluated challenge, Statements 4 and 10 evaluated

tension, Statements 6 and 8 evaluated positive affects

and Statements 9 and 11 evaluated negative affects.

Similarly, each module from Part 3 offered four dif-

ferent statements. Statement 1 evaluated enjoyment,

Statement 2 ease of use, Statement 3 sense of control

and Statement 4 evaluated preference, respectively. A

total of 52 statements were presented to participants

in the survey. To answer the survey statements, partic-

ipants used a Likert scale to determine the level of the

agreement they had with them. Value one represented

the lowest level of agreement, while ﬁve represented

the highest one. Following the evaluation guidelines

offered in the Game Experience Questionnaire (IJs-

selsteijn et al., 2013), each component was analyzed

by directly comparing the average answer score be-

tween the interaction devices.

4.2.1 Pentomino Puzzle Level

The average survey results for the Pentomino puzzle

level are shown in Table 1. Overall, the HTC Vive

controller was perceived to offer a better experience

in the Pentomino puzzle level when picking, dropping

and rotating pieces, compared with the Leap Motion

sensor. In the level of challenge experienced when

using the interaction devices, the Leap Motion sensor

was perceived to be the more challenging. The ten-

sion generated by the use of the interaction devices

was perceived to be lower on the HTC Vive controller,

while the perceived positive affects were higher on the

same device. Despite offering a better-perceived ex-

perience on all previous components, the negative af-

fects results were not considerable different between

the HTC Vive controller and the Leap Motion sensor.

4.2.2 Ball-throwing Level

The average survey results for the Ball-throwing level

survey are shown in Table 2. As an overview, par-

Table 1: Average scores for the experience components in

the Pentomino puzzle level.

Component Vive Controller Leap Motion

Competence 4 2.85

Challenge 1.62 2.75

Tension 1.45 2.27

Positive Affect 4.27 3.61

Negative Affect 1.12 1.27

Table 2: Average scores for the experience components in

the Ball-throwing level.

Component Vive Controller Leap Motion

Competence 3.73 2.86

Challenge 1.72 3.05

Tension 1.37 2.2

Positive Affect 4.02 3.38

Negative Affect 1.17 1.57

Table 3: Average scores for the experience comparison

components between the HTC Vive controller and the Leap

Motion sensor.

Component Vive Controller Leap Motion

Enjoyment 4.25 4.35

Ease of Use 4.5 3.15

Sense of Control 4.35 2.95

Preference 3.95 2.55

ticipants perceived to have a better experience using

the HTC Vive controller than the Leap Motion sen-

sor. They felt to be more competent at picking up and

throwing balls when using the HTC Vive controller

and they gave a higher positive affect score when in-

teracting with it. The Leap Motion sensor was per-

ceived to generate more tension and challenge with

its use. Despite this, the negative affects perceived by

players were not considerably different between the

devices.

4.2.3 Experience between Devices

The average scores for the experience comparison be-

tween the interaction devices can be seen in Table 3.

Overall, the HTC Vive controller was perceived to be

the better option among the interaction devices. It

was considered by participants easier to use than the

Leap Motion sensor, offering a better sense of con-

trol over the evaluated interaction techniques, and be-

ing the preferred interaction device for the proposed

tasks. Nonetheless, despite having lower performance

and a worse perceived experience than the HTC Vive

controller, the Leap Motion sensor was reported as the

most enjoyable interaction device to use in the study

by a small margin.

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

108

5 DISCUSSION

Results showed that participants had an overall bet-

ter performance and experience when using the HTC

Vive controller, as predicted in the hypothesis. Nev-

ertheless, the considerable differences in the perfor-

mance and experience results were unexpected.

In the Pentomino puzzle level, participants were

able to pick up, rotate and drop objects faster with

the HTC Vive controller, requiring on average a lower

number of pieces grabbed with this device than with

the Leap Motion sensor. A possible explanation for

this could be the way the devices triggered the inter-

action techniques, which were activated by only using

the trigger button in the HTC Vive controller, offer-

ing a simple execution with an immediate response

from the game engine. However, for the Leap Motion

sensor, a more robust calculation was needed to de-

termine the movements, positions and collisions from

different parts of the hand, making it more susceptible

to errors that affected the accuracy of the sensor. Ad-

ditionally, it was observed during initial tests that sub-

tle and slow ﬁnger movements were nor effective for

triggering the interaction techniques, especially drop-

ping objects, which showed a ”sticky hand” effect

when performed in this manner. To control for this

phenomenon, fast and exaggerated movements were

instructed to participants during the tutorial level.

For the Ball-throwing level, the Leap Motion sen-

sor had a worse overall performance among the de-

vices. In addition to the issues previously exposed,

the limited tracking capabilities of the Leap Motion

sensor had a detrimental effect in this level. Thanks to

the constant tracking from the base stations and sim-

ple execution of the throwing interaction, movements

performed by the HTC Vive controller were smooth

and consistent among participants. Since the interac-

tion area for the Leap Motion is smaller and tracking

is only possible when the hand is in front of the de-

vice, a natural throw movement was not possible with

the proposed setup. When a ball was grabbed, par-

ticipants instinctively raised and moved their hand to-

wards the back of their head to charge a throw. Since

the Leap Motion sensor was attached to the front of

the headset, their hands left the interaction area caus-

ing erratic movements and abruptly readjusting the

virtual elements once the hand re-entered it. Partic-

ipants were informed about this limitation during the

tutorial level and were instructed in how to perform

the throwing interaction properly, making sure that

the hand was always within the interaction area and

that the hand dropping gesture was fast and exagger-

ated. Nevertheless and despite the training offered in

the tutorial level, some participants, instinctively, per-

formed the natural throwing movement affecting the

performance data.

Results from Part 1 and 2 of the survey showed

that participants had an overall better experience

when using the HTC Vive controller than the Leap

Motion sensor. The sense of competence and posi-

tive affects experienced by participants were higher

when using the HTC Vive controller, probably due a

more efﬁcient activation of the interaction techniques

and better tracking capabilities when compared to

the Leap Motion sensor. These same reasons might

have affected the perceived level of challenge and

tension, since the Leap Motion sensor scored higher

in those components, especially in the Ball-throwing

level. Results from Part 3 of the survey showed that

the HTC Vive controller was perceived to be easier to

use than the Leap Motion sensor, offering better con-

trol over the proposed tasks in this study and was the

preferred device for this evaluation. Nevertheless and

despite having a lower score in the majority of the sur-

vey questions, the difference between the Leap Mo-

tion sensor and the HTC Vive controller, regarding the

Negative affects and user enjoyment components, was

reasonably low. This particular result suggests that

even if the interaction capabilities of the Leap Motion

sensor were not on par with the HTC Vive controller,

the use NUIs generated a rather strong impression in

the participants, as some of them mentioned. Having

the ability to interact with virtual environments by us-

ing your own body, is a novel interaction concept that

has rarely been experienced before, and that could

have the potential of redeﬁning the way we commu-

nicate with VR applications. This could be the reason

why, despite having the lower performance and expe-

rience scores, the Leap Motion sensor reported higher

scores for the enjoyment component.

6 CONCLUSION AND FUTURE

WORK

The study presented in this paper has offered an anal-

ysis of the variation in performance and experience

between the HTC Vive controller and the Leap Mo-

tion sensor when used for VR game interaction tasks.

A user study exposed 20 participants to a couple of

VR games, each composed by a Pentomino puzzle

and Ball-throwing level. Performance data was cap-

tured automatically by the games while the perceived

experience was evaluated through a survey based on a

modiﬁed version of the Game Experience Question-

naire (IJsselsteijn et al., 2013). Results showed that

the HTC Vive controller offered overall better per-

formance when compared to the Leap Motion sensor.

Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor

109

More straightforward interaction activation and better

tracking capabilities allowed participants to achieve

higher scores when using the HTC Vive interaction

device. Similarly, the HTC Vive controller was also

considered to offer a better overall experience to par-

ticipants regarding the ease of use, sense of control

and user preference. Nevertheless, the survey results

for the Leap Motion sensor showed little difference in

the scores for negative affects, when compared with

the HTC Vive and, despite interactive and tracking

limitations, it reported the higher scores for enjoy-

ment in this study.

Future work could evaluate the capabilities of

the Leap Motion sensor in other game genres, since

games this study were based on the physical inter-

action between virtual elements. Board, adventure

or ﬁghting games are considered interesting environ-

ments for the design and implementation of NUIs.

Also, the large difference seen in the performance

results motivates a more detailed examination to de-

termine how the interaction devices and techniques

could have led to such results. A further analysis re-

lating the amount of grabs per piece, or the amount

of actions (grabs) per time interval could offer a bet-

ter insight of how and when the complexity and accu-

racy varied among the devices. Additionally, different

software solutions could be explored to compensate

for the limitations found with the Leap Motion sensor.

Improved gesture control and recognition, snapping

capabilities and physiologically aware environments

are possible ideas to explore in the future.

ACKNOWLEDGEMENTS

This work was supported in part by KK-stiftelsen

Sweden, through the ViaTecH Synergy Project (con-

tract 20170056). We thank all volunteers, staff at

DIKR for feedback on the research. The study has

been granted ethical approval (dnr: 2018/624).

REFERENCES

Bachmann, D., Weichert, F., and Rinkenauer, G. (2018).

Review of Three-Dimensional Human-Computer In-

teraction with Focus on the Leap Motion Controller.

Sensors, 18(7):2194.

Caggianese, G., Gallo, L., and Neroni, P. (2016). An Inves-

tigation of Leap Motion Based 3d Manipulation Tech-

niques for Use in Egocentric Viewpoint. In De Pao-

lis, L. T. and Mongelli, A., editors, Augmented Real-

ity, Virtual Reality, and Computer Graphics, volume

9769, pages 318–330. Springer International Publish-

ing, Cham.

Caggianese, G., Gallo, L., and Neroni, P. (2019). The Vive

Controllers vs. Leap Motion for Interactions in Vir-

tual Environments: A Comparative Evaluation. In

De Pietro, G., Gallo, L., Howlett, R. J., Jain, L. C.,

and Vlacic, L., editors, Intelligent Interactive Multi-

media Systems and Services, volume 98, pages 24–33.

Springer International Publishing, Cham.

Erra, U., Malandrino, D., and Pepe, L. (2018). Virtual Real-

ity Interfaces for Interacting with Three-Dimensional

Graphs. International Journal of HumanComputer In-

teraction, pages 1–14.

Gusai, E., Bassano, C., Solari, F., and Chessa, M. (2017).

Interaction in an Immersive Collaborative Virtual Re-

ality Environment: A Comparison Between Leap Mo-

tion and HTC Controllers. In Battiato, S., Farinella,

G. M., Leo, M., and Gallo, G., editors, New Trends in

Image Analysis and Processing ICIAP 2017, volume

10590, pages 290–300. Springer International Pub-

lishing, Cham.

Hornsey, R. L. and Hibbard, P. B. (2015). Evaluation of

the accuracy of the Leap Motion controller for mea-

surements of grip aperture. In Proceedings of the

12th European Conference on Visual Media Produc-

tion - CVMP ’15, pages 1–1, London, United King-

dom. ACM Press.

IJsselsteijn, W., de Kort, Y., and Poels, K. (2013). The

Game Experience Questionnaire. Technische Univer-

siteit Eindhoven.

Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., and

Tachibana, K. (2000). Virtual object manipulation on

a table-top AR environment. In Proceedings IEEE and

ACM International Symposium on Augmented Real-

ity (ISAR 2000), pages 111–119, Munich, Germany.

IEEE.

Lee, S., Park, K., Lee, J., and Kim, K. (2017). User Study

of VR Basic Controller and Data Glove as Hand Ges-

ture Inputs in VR Games. In 2017 International Sym-

posium on Ubiquitous Virtual Reality (ISUVR), pages

1–3, Nara, Japan. IEEE.

Lopez, F., Navarro, D., and Sundstedt, V. (2017). Ethi-

cal Considerations for the Use of Virtual Reality: An

Evaluation of Practices in Academia and Industry. In

Lindeman, R. W., Bruder, G., and Iwai, D., editors,

ICAT-EGVE 2017 - International Conference on Arti-

ﬁcial Reality and Telexistence and Eurographics Sym-

posium on Virtual Environments. The Eurographics

Association.

McMahan, R. P., Alon, A. J. D., Lazem, S., Beaton, R. J.,

Machaj, D., Schaefer, M., Silva, M. G., Leal, A.,

Hagan, R., and Bowman, D. A. (2010). Evaluat-

ing natural interaction techniques in video games. In

2010 IEEE Symposium on 3D User Interfaces (3DUI),

pages 11–14, Waltham, MA, USA. IEEE.

Peter Wozniak, Oliver Vauderwange, A. M. N. J. D. C.

(2016). Possible applications of the leap motion con-

troller for more interactive simulated experiments in

augmented or virtual reality.

Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D.

(2013). Analysis of the Accuracy and Robustness of

the Leap Motion Controller. Sensors, 13(5):6380–

6393.

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

110