Evaluating Player Performance and Experience in Virtual Reality Game
Interactions using the HTC Vive Controller and Leap Motion Sensor
Diego Navarro and Veronica Sundstedt
Blekinge Institute of Technology, Karlskrona, Sweden
Keywords:
Game Interaction, HTC Vive, Leap Motion, Virtual Reality, User Study, Performance, Experience.
Abstract:
An important aspect of virtual reality (VR) interfaces are novel natural user interactions (NUIs). The increased
use of VR games requires the evaluation of novel interaction techniques that allow efficient manipulations of
3D elements using the hands of the player. Examples of VR devices that support these interactions include the
HTC Vive controller and the Leap Motion sensor. This paper presents a quantitative and qualitative evalua-
tion of player performance and experience in a controlled experiment with 20 volunteering participants. The
experiment evaluated the HTC Vive controller and the Leap Motion sensor when manipulating 3D objects in
two VR games. The first game was a Pentomino puzzle and the second game consisted of a ball-throwing
task. Four interaction techniques (picking up, dropping, rotating, and throwing objects) were evaluated as part
of the experiment. The number of user interactions with the Pentomino pieces, the number of ball throws, and
game completion time were metrics used to analyze the player performance. A questionnaire was also used
to evaluate the player experience regarding enjoyment, ease of use, sense of control and user preference. The
overall results show that there was a significant decrease in player performance when using the Leap Motion
sensor for the VR game tasks. Participants also reported that hand gestures with the Leap Motion sensor were
not as reliable as the HTC Vive controller. However, the survey showed positive responses when using both
technologies. The paper also offers ideas to keep exploring the capabilities of NUI techniques in the future.
1 INTRODUCTION
Virtual reality (VR) has become a popular consumer
technology thanks to its mass production and constant
development of novel interactive content tailored for
it. VR offers a high level of immersion due, in part,
to its coverage of the full peripheral view of users
and head tracking capabilities. Despite this, the ini-
tial level of immersion experienced with VR tech-
nology can be deteriorated when users fail to inter-
act naturally and effectively with the virtual environ-
ments (Kato et al., 2000; Lee et al., 2017; Bach-
mann et al., 2018). Hardware manufacturers, includ-
ing HTC, have addressed this issue by implement-
ing direct hand manipulation through a set of propri-
etary peripherals, adapting design concepts from tra-
ditional gamepad controllers (buttons, triggers, rum-
ble packs and wireless connectivity) into hand-held
devices suitable for 3D-space interaction. The HTC
Vive controller is one example of them. More recent
designs have tried to further enhance the user experi-
ence by introducing natural user interactions (NUIs),
techniques that recreate real-life action within a cer-
tain level of fidelity (McMahan et al., 2010), as a
novel method for interacting with VR environments.
An example of them is the Leap Motion sensor, which
allows hand tracking and natural hand manipulation
of virtual objects. Different VR interactive applica-
tions have featured novel interaction techniques using
the HTC Vive controller and the Leap Motion sen-
sor. Nonetheless, despite previous contributions, few
studies offer a detailed evaluation of how user per-
formance and experience could be affected by their
use. This situation offers an opportunity to analyze
how efficient NUIs could be when performing basic
interactive tasks in VR environments, in comparison
to hand-held devices. To contribute to the previous
gap, the following study proposes a couple of research
questions: (1) What are the variations in player per-
formance between the HTC Vive controller and the
Leap Motion sensor when used as the main inter-
action input for picking up, dropping, rotating and
throwing objects in VR games? and (2) What are
the variations in the perceived player experience be-
tween the HTC Vive controller and the Leap Motion
sensor when analyzed in terms of enjoyment, ease
Navarro, D. and Sundstedt, V.
Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor.
DOI: 10.5220/0007362401030110
In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), pages 103-110
ISBN: 978-989-758-354-4
Copyright
c
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
103
of use, sense of control and user preference?. An
initial hypothesis proposes that the HTC Vive con-
troller will offer a better overall performance result,
compared with the Leap Motion sensor. However,
there will not be a significantly large difference be-
tween the results obtained with both sensors and the
Leap motion will be the preferred one by participants.
The previous questions are addressed by performing
a user study, that differentiates from similar contri-
butions (e.g. (Caggianese et al., 2019)) by offering
a more complex game environment in which track-
ing precision and accurate control over the interaction
techniques play a greater factor in the performance
measurements. Results from this study could help to
identify some of the possibilities and challenges, of-
fered by NUIs and hand-held controllers, in the field
of efficient 3D-interaction techniques. Additionally,
it can provide an initial understanding of how these
technologies can affect the overall perceived user ex-
perience when used in VR environments and other vi-
sual and interactive applications.
2 BACKGROUND
The HTC Vive controller is a hand-held motion and
pointing device that resembles a television remote
controller with a hollow ring attached to its top. It
features a track-pad in the front, a couple of buttons
on either side and a trigger button in the back side
of its handle. It uses a set of 24 infra-red (IR) sen-
sors for positional tracking with the HTC Vive base
stations (devices responsible for the tracking of the
headset and controllers), and operates in a frequency
ranging between 250Hz and 1KHz
1
. On the other
hand, the Leap Motion sensor is a small rectangular
device that combines three sources of IR light, to-
gether with a stereoscopic array of IR cameras, fitted
inside of a metallic frame
2
. It allows the tracking of
hand movements, with a 5-finger level of detail, with-
out the need of wearing any additional hardware. It
operates at a frequency of 200 frames/sec, with a field
of view of 150
, an interaction area of 0.74m
2
approx-
imately and an overall accuracy of 0.7mm (Weichert
et al., 2013; Hornsey and Hibbard, 2015). Previous
contributions analyzing VR technologies, in combi-
nation with NUI devices, have focused on exploring
their interactive capabilities in a wide variety of sce-
narios, while also identifying some of their poten-
tial drawbacks and limitations (Peter Wozniak, 2016).
1
https://www.vrheads.com/exposing-magic-behind-htc-
vive-controller, visited October 2018.
2
https://learn.sparkfun.com/tutorials/leap-motion-
teardown, visited October 2018.
Nonetheless, only a small set of recent studies offer
a direct comparison between the HTC Vive controller
and the Leap Motion sensor, evaluating quantitative
and qualitative variables from the user performance
and experience, that derives from their respective use.
An initial usability evaluation analyzed two dif-
ferent interaction techniques for manipulating objects
in a VR environment. The techniques focused on
translating and rotating virtual objects through direct
(touching the object) and constrained (using a visual
pivot attached to the object) manipulation. The results
were qualitatively analyzed and showed that partici-
pants preferred to use direct manipulation for trans-
lating objects, while constrained manipulation was
the preferred method for rotation (Caggianese et al.,
2016).
A performance analysis was conducted on a col-
laborative virtual environment. In this study, pulling
and pushing interactions were tested by 30 partici-
pants in an exergame setup. Two participants took the
test at a time. The first one was immersed in the vir-
tual environment while the second monitored the per-
formance of the first participant and provided visual
assistance to complete the task. Performance results
were analyzed quantitatively and showed an overall
better performance of the HTC Vive controller over
the Leap Motion sensor, requiring a less amount of
interactions and less time to complete the experimen-
tal task (Gusai et al., 2017).
A similar study evaluated the variations in user ex-
perience when interactively manipulating 3D graphs,
comparing traditional input methods (gamepads,
mouse and keyboard) against natural interaction tech-
niques with the Leap Motion sensor. Results have
shown that participants had an overall better experi-
ence when using the traditional input methods, sug-
gesting ease of use and responsive control to be some
of the determining factors for such an outcome. Even
if participants found the use of natural interaction
techniques interesting and fun, the limitations of the
technology and the variations in user preference for
performing and articulating gestures made the Leap
Motion sensor a challenging platform to manipulate
3D graphs (Erra et al., 2018).
Finally, a more recent study analyzed the possi-
bilities each device offered in terms of interaction
design, and evaluated their respective performance
when used for object manipulation in virtual environ-
ments. Results from this study show an overall user
preference for the HTC Vive controller, while identi-
fying a necessity for simplifying complex interaction
techniques in virtual environments, to deliver an over-
all better user experience (Caggianese et al., 2019).
HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications
104
3 METHODOLOGY
A user study was selected as the main methodology
for the research. Volunteering participants were ex-
posed to two different VR games featuring either the
HTC Vive controller or the Leap Motion sensor. Dur-
ing gameplay, different data was captured as quantita-
tive metrics for evaluating the player performance. A
survey, consisting of a modified version of the Game
Experience Questionnaire (IJsselsteijn et al., 2013),
was presented to the participants in order to ana-
lyze the perceived player experience upon completing
each game level.
3.1 Participant Criteria
The participant criteria aimed to provide a more ho-
mogeneous group in terms of age, previous experi-
ences with VR and the evaluated game genres. It was:
(1) experience with VR content of at least 2 hours, (2)
experience using the HTC Vive controller of at least 2
hours, (3) experience with puzzle games (e.g. Tetris,
World of Goo, Antichamber or Lyne) of at least 10
hours, (4) an age range between 20 and 28 years old,
(5) no previous cases of photo-sensitivity or epilepsy
and (6) no other medical or physiological condition
that limits the use of stereo vision or the use of hands
and fingers. Only participants that self-reported they
fulfilled these criteria were included.
3.2 Experiment Procedure
The experimentation was a within-participant user
study, with one factor and two levels of repeated mea-
sures. Volunteers were initially presented with a con-
sent form, describing the inclusion criteria, goal of
the study, experimental procedure, stimuli, tasks, re-
quired time, risk of participation, data captured and
data protection policies. Upon agreeing to and sign-
ing the form, the experimenter then verbally intro-
duced the two interaction devices used for the test and
guided the participant to the gameplay area (see Sec-
tion 3.3). The HTC Vive headset was placed upon the
participant head and, since the interaction devices are
capable of tracking both hands, participants were told
the study should be performed using only their dom-
inant one. A couple of VR games were presented to
the participants as the main stimuli. One game fea-
tured the HTC Vive controller as the primary inter-
action device, while the other used the Leap Motion
sensor. The VR games, composed by a tutorial, a Pen-
tomino puzzle level and a Ball-throwing level (shown
in Figure 1 and Figure 2), were identical in content,
being the interaction devices they featured, together
with their respective interaction techniques, and the
reference image provided for the Pentomino level the
only variations between them. Both games were de-
veloped using the Unity game engine, v. 2018.2.7f1.
The HTC Vive headset and controller were integrated
using the Steam VR plugin for Unity, v. 2.0.1, by
Valve Corp. For controlling the Leap Motion sensor,
the Unity Assets for Leap Motion Orion Beta v. 4.4.0
and the Leap Motion Interaction Engine Module for
Unity v. 1.2.0, both published by Leap Motion Inc,
were used.
The tutorial level was presented first and intro-
duced the participants to the virtual environment, the
3D elements and the four main interaction techniques
evaluated in the study: picking up, rotating, dropping
and throwing virtual objects. Participants were ver-
bally guided through the tutorial level, receiving ex-
planations about the rules of the game, experimental
conditions, tasks and how to perform them correctly
using the respective interaction device. Participants
were allowed to proceed to the following levels only
when they manifested a complete understanding and
sense of control over the game tasks and how to use
the interaction devices. In this way, the lack of expe-
rience with the game or unfamiliarity with the inter-
action techniques were minimized.
The Pentomino puzzle was the first game level
participants were exposed to, being tasked with as-
sembling a diamond-shaped figure by picking, drop-
ping and rotating a set of 12 Pentomino pieces (ge-
ometric shapes composed of five cubes of the same
size, connected through one or several faces
3
) as
fast as possible. A reference image was offered as
a visual aid, showing the correct arrangement of the
pieces. The same diamond shape was used in both
games, but the original reference image was flipped
horizontally and vertically in the Leap Motion game
(see inset in Figure 1) to control for carryover effects
without the risk of increasing the difficulty of the task.
The Ball-throwing level was shown last, tasking play-
ers with hitting six targets, floating in the horizon, by
throwing a set of three balls at them. The task had
to be completed in the least amount of time possible.
Targets were placed at different distances and heights
from the initial viewpoint of the participants, while
the balls appeared directly in front of them. Players
were surrounded by leaned walls with no friction, so
balls could rapidly bounce back to the their position.
To grab a piece with the HTC Vive controller, par-
ticipants needed to press and hold the trigger button
while touching its surface with the top ring of the
controller. Releasing the trigger button dropped the
piece. For rotating, participants needed to physically
3
https://en.wikipedia.org/wiki/Pentomino
Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor
105
Figure 1: The Pentomino puzzle level, using the HTC Vive
controller. Inset in the lower-right corner shows the refer-
ence images used for the Pentomino levels.
Figure 2: The Ball-throwing level, using the Leap Motion
sensor. Inset in the lower-right corner shows the HTC Vive,
its controller and the Leap Motion sensor attached to the
front of the headset.
rotate the controller while holding the piece in their
hands. To perform these actions with the Leap Motion
sensor, participants needed to recreate similar hand
movements as they would usually do in the real world.
To grab a piece, the thumb and any other finger from
the participants’ hand needed to be in contact with its
surface. Participants were instructed to grab pieces
by pinching or claw-gripping them to avoid tracking
issues with the sensor since a closed fist showed an er-
ratic behavior during initial tests. To release a piece,
participants simply opened their hand widely. To ro-
tate a piece, participants needed to hold it in their hand
while rotating it. For throwing a ball, participants had
to release it while performing the throwing motion.
Upon completing each of the VR games, the head-
set was removed from the participants, and a modified
version of the Game Experience Questionnaire (IJs-
selsteijn et al., 2013) was presented. The survey
evaluated their perceived experience when interacting
with the virtual world, by performing each of the eval-
uated tasks with the respective interaction device. The
game stimuli were presented counterbalanced to the
participants in an alternated order. This means that
if the current participant started with the game fea-
turing the HTC Vive controller, the next started with
the game using the Leap Motion sensor. When both
games were completed, the final part of the survey
was given. The final part evaluated their overall opin-
ion and preferences when having a general compari-
son between both devices.
3.3 Experimental Setup and Equipment
The study was conducted within a dedicated con-
trolled environment, provided by the Department of
Creative Technologies (DIKR) at the Blekinge Insti-
tute of Technology. Following the criteria for an ap-
propriate exposure of VR content proposed by (Lopez
et al., 2017), a dedicated squared gameplay area of 2.5
x 2.5 meters, free of any obstacles, was prepared for
the participants. Also, participants were seated during
the entire length of the experimentation.
Outside the gameplay area, a dedicated worksta-
tion computer ran the VR games while, simultane-
ously, managing the different apparatus needed for
the experimentation. The computer had an Intel Core
i7 6700k CPU @ 4.00GHz, a Nvidia GeForce GTX
980 GPU and 16 GB of Corsair DDR4 RAM @
1333MHz. The apparatus used for this study were the
HTC Vive VR headset, its base stations, controller,
and the Leap Motion sensor. The HTC Vive controller
was held in the participants’ hands and was attached
to their wrists. The Leap Motion sensor was attached
to the headset using the Universal VR Dev Mount that
was glued to the front of it (see inset in Figure 2).
4 RESULTS
A total of 20 participants volunteered for the study.
Gathered data were classified into performance (Sec-
tion 4.1) and experience results (Section 4.2). Sta-
tistical significance tests were applied to the gathered
performance data, while the survey results were an-
alyzed according to the scoring parameters exposed
in the Game Experience Questionnaire (IJsselsteijn
et al., 2013).
4.1 Performance Results
The total amount of pieces grabs needed to com-
plete the game, together with the total completion
time, were the evaluation metrics for the performance
analysis in the Pentomino level. The Ball-throwing
level evaluated the total number of throws required
to finish the level, along with the completion time.
Performance data was initially tested to verify the
ANOVA assumptions and determine an appropriate
significance test. Normality was evaluated using
HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications
106
Pentomino Puzzle Level Results
0
20
40
60
80
100
120
140
160
180
200
220
240
260
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Grabs
Participant
(A) Amount of Pieces Grabbed
HTC Vive
Leap Motion
0
20
40
60
80
100
120
140
160
HTC Vive Leap Motion
Grabs
Device
(B) Average Grabs
0
60
120
180
240
300
360
420
480
540
600
660
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Seconds
Participant
(C) Completion Time
HTC Vive
Leap Motion
0
60
120
180
240
300
360
420
HTC Vive Leap Motion
Seconds
Device
(D) Average Completion Time
Figure 3: Performance summary for the Pentomino puzzle level results. Figure (A) shows the total amount of pieces grabs per
participant and device. Figure (B) shows the average number of grabs per device. Figure (C) shows the total level completion
time per participant and device. Figure (D) shows the average completion time per device.
Ball-Throwing Level Results
0
10
20
30
40
50
60
70
80
90
100
110
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Throws
Participant
(A) Balls Thrown
HTC Vive
Leap Motion
0
5
10
15
20
25
30
35
40
45
50
55
HTC Vive Leap Motion
Throws
Device
(B) Average Throws
0
30
60
90
120
150
180
210
240
270
300
330
360
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Seconds
Participant
(C) Completion Time
HTC Vive
Leap Motion
0
30
60
90
120
150
180
HTC Vive Leap Motion
Seconds
Device
(D) Average Completion Time
Figure 4: Performance summary for the Ball-throwing level results. Figure (A) shows the total amount of throws per partici-
pant and device. Figure (B) shows the average number of throws per device. Figure (C) shows the total level completion time
per participant and device. Figure (D) shows the average completion time per device.
the Shapiro-Wilk test on data residuals, while ho-
moscedasticity was reviewed with the Levene’s test.
None of the gathered data was able to satisfy the
ANOVA assumptions. Therefore a non-parametric
test was applied. Based on the data properties and
the proposed experimental design, a Wilcoxon-Mann-
Withney (WMW) test was used for the statistical sig-
nificance evaluation.
4.1.1 Pentomino Puzzle Level
The summary of the results obtained in the Pentomino
puzzle level can be seen in Figure 3. Overall, partic-
ipants had better performance when using the HTC
Vive controller, requiring fewer pieces grabs and time
to complete the level. For the pieces grabs, partici-
pants averaged 50.1 when using the HTC Vive con-
troller, compared to 91.35 with the Leap Motion sen-
sor. After applying the WMW test, results showed
a statistically significant difference between the HTC
Vive controller and the Leap Motion sensor.
W MW
(1,n=20)
= 3.7355, p < 0.05
For completion times, participants required more
time to complete the game using the Leap Motion
sensor, scoring an average of 251.48 seconds. The
time needed with the HTC Vive controller was less,
averaging 135.29 seconds. Results from the WMW
test showed a statistically significant difference in the
completion times for each interaction device.
W MW
(1,n=20)
= 5.1395, p < 0.05
4.1.2 Ball-throwing Level
A summary of the results gathered from the Ball-
throwing level is illustrated in Figure 4. Participants
had overall better performance when using the HTC
Vive controller, needing fewer throws and less time to
complete the game, when compared with the scores
obtained with the Leap Motion sensor. The average
amount of throws needed to complete the level us-
ing the HTC Vive controller was 10.1, while 30.45
was the average needed with the Leap Motion sensor.
The WMW test showed a statistically significant dif-
ference between the number of throws needed with
each interaction device.
W MW
(1,n=20)
= 4.3283, p < 0.05
The time required to finish the level was higher
when using the Leap Motion sensor, with an average
of 107.73 seconds among the participants, when com-
pared with the average time of 33.61 seconds obtained
with the HTC Vive controller. Time results had a
statistically significant difference between them when
evaluated with the WMW test.
W MW
(1,n=20)
= 4.0846, p < 0.05
Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor
107
4.2 Experience Results
The survey was composed by three parts: Part 1 eval-
uated the game featuring the HTC Vive controller,
Part 2 focused on the game using the Leap Motion
sensor and Part 3 made a direct comparison between
both interaction devices. Each part of the survey was,
subsequently, composed of two modules. For Part 1
and 2, there was a module for the Pentomino puzzle
and the ball throwing levels. In Part 3, each interac-
tion device had its module respectively. Each module
from Part 1 and Part 2 presented 11 different state-
ments to participants. They evaluated five compo-
nents from the perceived experience: Statements 1,
3 and 5 evaluated competence, Statements 2 and 7
evaluated challenge, Statements 4 and 10 evaluated
tension, Statements 6 and 8 evaluated positive affects
and Statements 9 and 11 evaluated negative affects.
Similarly, each module from Part 3 offered four dif-
ferent statements. Statement 1 evaluated enjoyment,
Statement 2 ease of use, Statement 3 sense of control
and Statement 4 evaluated preference, respectively. A
total of 52 statements were presented to participants
in the survey. To answer the survey statements, partic-
ipants used a Likert scale to determine the level of the
agreement they had with them. Value one represented
the lowest level of agreement, while five represented
the highest one. Following the evaluation guidelines
offered in the Game Experience Questionnaire (IJs-
selsteijn et al., 2013), each component was analyzed
by directly comparing the average answer score be-
tween the interaction devices.
4.2.1 Pentomino Puzzle Level
The average survey results for the Pentomino puzzle
level are shown in Table 1. Overall, the HTC Vive
controller was perceived to offer a better experience
in the Pentomino puzzle level when picking, dropping
and rotating pieces, compared with the Leap Motion
sensor. In the level of challenge experienced when
using the interaction devices, the Leap Motion sensor
was perceived to be the more challenging. The ten-
sion generated by the use of the interaction devices
was perceived to be lower on the HTC Vive controller,
while the perceived positive affects were higher on the
same device. Despite offering a better-perceived ex-
perience on all previous components, the negative af-
fects results were not considerable different between
the HTC Vive controller and the Leap Motion sensor.
4.2.2 Ball-throwing Level
The average survey results for the Ball-throwing level
survey are shown in Table 2. As an overview, par-
Table 1: Average scores for the experience components in
the Pentomino puzzle level.
Component Vive Controller Leap Motion
Competence 4 2.85
Challenge 1.62 2.75
Tension 1.45 2.27
Positive Affect 4.27 3.61
Negative Affect 1.12 1.27
Table 2: Average scores for the experience components in
the Ball-throwing level.
Component Vive Controller Leap Motion
Competence 3.73 2.86
Challenge 1.72 3.05
Tension 1.37 2.2
Positive Affect 4.02 3.38
Negative Affect 1.17 1.57
Table 3: Average scores for the experience comparison
components between the HTC Vive controller and the Leap
Motion sensor.
Component Vive Controller Leap Motion
Enjoyment 4.25 4.35
Ease of Use 4.5 3.15
Sense of Control 4.35 2.95
Preference 3.95 2.55
ticipants perceived to have a better experience using
the HTC Vive controller than the Leap Motion sen-
sor. They felt to be more competent at picking up and
throwing balls when using the HTC Vive controller
and they gave a higher positive affect score when in-
teracting with it. The Leap Motion sensor was per-
ceived to generate more tension and challenge with
its use. Despite this, the negative affects perceived by
players were not considerably different between the
devices.
4.2.3 Experience between Devices
The average scores for the experience comparison be-
tween the interaction devices can be seen in Table 3.
Overall, the HTC Vive controller was perceived to be
the better option among the interaction devices. It
was considered by participants easier to use than the
Leap Motion sensor, offering a better sense of con-
trol over the evaluated interaction techniques, and be-
ing the preferred interaction device for the proposed
tasks. Nonetheless, despite having lower performance
and a worse perceived experience than the HTC Vive
controller, the Leap Motion sensor was reported as the
most enjoyable interaction device to use in the study
by a small margin.
HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications
108
5 DISCUSSION
Results showed that participants had an overall bet-
ter performance and experience when using the HTC
Vive controller, as predicted in the hypothesis. Nev-
ertheless, the considerable differences in the perfor-
mance and experience results were unexpected.
In the Pentomino puzzle level, participants were
able to pick up, rotate and drop objects faster with
the HTC Vive controller, requiring on average a lower
number of pieces grabbed with this device than with
the Leap Motion sensor. A possible explanation for
this could be the way the devices triggered the inter-
action techniques, which were activated by only using
the trigger button in the HTC Vive controller, offer-
ing a simple execution with an immediate response
from the game engine. However, for the Leap Motion
sensor, a more robust calculation was needed to de-
termine the movements, positions and collisions from
different parts of the hand, making it more susceptible
to errors that affected the accuracy of the sensor. Ad-
ditionally, it was observed during initial tests that sub-
tle and slow finger movements were nor effective for
triggering the interaction techniques, especially drop-
ping objects, which showed a ”sticky hand” effect
when performed in this manner. To control for this
phenomenon, fast and exaggerated movements were
instructed to participants during the tutorial level.
For the Ball-throwing level, the Leap Motion sen-
sor had a worse overall performance among the de-
vices. In addition to the issues previously exposed,
the limited tracking capabilities of the Leap Motion
sensor had a detrimental effect in this level. Thanks to
the constant tracking from the base stations and sim-
ple execution of the throwing interaction, movements
performed by the HTC Vive controller were smooth
and consistent among participants. Since the interac-
tion area for the Leap Motion is smaller and tracking
is only possible when the hand is in front of the de-
vice, a natural throw movement was not possible with
the proposed setup. When a ball was grabbed, par-
ticipants instinctively raised and moved their hand to-
wards the back of their head to charge a throw. Since
the Leap Motion sensor was attached to the front of
the headset, their hands left the interaction area caus-
ing erratic movements and abruptly readjusting the
virtual elements once the hand re-entered it. Partic-
ipants were informed about this limitation during the
tutorial level and were instructed in how to perform
the throwing interaction properly, making sure that
the hand was always within the interaction area and
that the hand dropping gesture was fast and exagger-
ated. Nevertheless and despite the training offered in
the tutorial level, some participants, instinctively, per-
formed the natural throwing movement affecting the
performance data.
Results from Part 1 and 2 of the survey showed
that participants had an overall better experience
when using the HTC Vive controller than the Leap
Motion sensor. The sense of competence and posi-
tive affects experienced by participants were higher
when using the HTC Vive controller, probably due a
more efficient activation of the interaction techniques
and better tracking capabilities when compared to
the Leap Motion sensor. These same reasons might
have affected the perceived level of challenge and
tension, since the Leap Motion sensor scored higher
in those components, especially in the Ball-throwing
level. Results from Part 3 of the survey showed that
the HTC Vive controller was perceived to be easier to
use than the Leap Motion sensor, offering better con-
trol over the proposed tasks in this study and was the
preferred device for this evaluation. Nevertheless and
despite having a lower score in the majority of the sur-
vey questions, the difference between the Leap Mo-
tion sensor and the HTC Vive controller, regarding the
Negative affects and user enjoyment components, was
reasonably low. This particular result suggests that
even if the interaction capabilities of the Leap Motion
sensor were not on par with the HTC Vive controller,
the use NUIs generated a rather strong impression in
the participants, as some of them mentioned. Having
the ability to interact with virtual environments by us-
ing your own body, is a novel interaction concept that
has rarely been experienced before, and that could
have the potential of redefining the way we commu-
nicate with VR applications. This could be the reason
why, despite having the lower performance and expe-
rience scores, the Leap Motion sensor reported higher
scores for the enjoyment component.
6 CONCLUSION AND FUTURE
WORK
The study presented in this paper has offered an anal-
ysis of the variation in performance and experience
between the HTC Vive controller and the Leap Mo-
tion sensor when used for VR game interaction tasks.
A user study exposed 20 participants to a couple of
VR games, each composed by a Pentomino puzzle
and Ball-throwing level. Performance data was cap-
tured automatically by the games while the perceived
experience was evaluated through a survey based on a
modified version of the Game Experience Question-
naire (IJsselsteijn et al., 2013). Results showed that
the HTC Vive controller offered overall better per-
formance when compared to the Leap Motion sensor.
Evaluating Player Performance and Experience in Virtual Reality Game Interactions using the HTC Vive Controller and Leap Motion Sensor
109
More straightforward interaction activation and better
tracking capabilities allowed participants to achieve
higher scores when using the HTC Vive interaction
device. Similarly, the HTC Vive controller was also
considered to offer a better overall experience to par-
ticipants regarding the ease of use, sense of control
and user preference. Nevertheless, the survey results
for the Leap Motion sensor showed little difference in
the scores for negative affects, when compared with
the HTC Vive and, despite interactive and tracking
limitations, it reported the higher scores for enjoy-
ment in this study.
Future work could evaluate the capabilities of
the Leap Motion sensor in other game genres, since
games this study were based on the physical inter-
action between virtual elements. Board, adventure
or fighting games are considered interesting environ-
ments for the design and implementation of NUIs.
Also, the large difference seen in the performance
results motivates a more detailed examination to de-
termine how the interaction devices and techniques
could have led to such results. A further analysis re-
lating the amount of grabs per piece, or the amount
of actions (grabs) per time interval could offer a bet-
ter insight of how and when the complexity and accu-
racy varied among the devices. Additionally, different
software solutions could be explored to compensate
for the limitations found with the Leap Motion sensor.
Improved gesture control and recognition, snapping
capabilities and physiologically aware environments
are possible ideas to explore in the future.
ACKNOWLEDGEMENTS
This work was supported in part by KK-stiftelsen
Sweden, through the ViaTecH Synergy Project (con-
tract 20170056). We thank all volunteers, staff at
DIKR for feedback on the research. The study has
been granted ethical approval (dnr: 2018/624).
REFERENCES
Bachmann, D., Weichert, F., and Rinkenauer, G. (2018).
Review of Three-Dimensional Human-Computer In-
teraction with Focus on the Leap Motion Controller.
Sensors, 18(7):2194.
Caggianese, G., Gallo, L., and Neroni, P. (2016). An Inves-
tigation of Leap Motion Based 3d Manipulation Tech-
niques for Use in Egocentric Viewpoint. In De Pao-
lis, L. T. and Mongelli, A., editors, Augmented Real-
ity, Virtual Reality, and Computer Graphics, volume
9769, pages 318–330. Springer International Publish-
ing, Cham.
Caggianese, G., Gallo, L., and Neroni, P. (2019). The Vive
Controllers vs. Leap Motion for Interactions in Vir-
tual Environments: A Comparative Evaluation. In
De Pietro, G., Gallo, L., Howlett, R. J., Jain, L. C.,
and Vlacic, L., editors, Intelligent Interactive Multi-
media Systems and Services, volume 98, pages 24–33.
Springer International Publishing, Cham.
Erra, U., Malandrino, D., and Pepe, L. (2018). Virtual Real-
ity Interfaces for Interacting with Three-Dimensional
Graphs. International Journal of HumanComputer In-
teraction, pages 1–14.
Gusai, E., Bassano, C., Solari, F., and Chessa, M. (2017).
Interaction in an Immersive Collaborative Virtual Re-
ality Environment: A Comparison Between Leap Mo-
tion and HTC Controllers. In Battiato, S., Farinella,
G. M., Leo, M., and Gallo, G., editors, New Trends in
Image Analysis and Processing ICIAP 2017, volume
10590, pages 290–300. Springer International Pub-
lishing, Cham.
Hornsey, R. L. and Hibbard, P. B. (2015). Evaluation of
the accuracy of the Leap Motion controller for mea-
surements of grip aperture. In Proceedings of the
12th European Conference on Visual Media Produc-
tion - CVMP ’15, pages 1–1, London, United King-
dom. ACM Press.
IJsselsteijn, W., de Kort, Y., and Poels, K. (2013). The
Game Experience Questionnaire. Technische Univer-
siteit Eindhoven.
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., and
Tachibana, K. (2000). Virtual object manipulation on
a table-top AR environment. In Proceedings IEEE and
ACM International Symposium on Augmented Real-
ity (ISAR 2000), pages 111–119, Munich, Germany.
IEEE.
Lee, S., Park, K., Lee, J., and Kim, K. (2017). User Study
of VR Basic Controller and Data Glove as Hand Ges-
ture Inputs in VR Games. In 2017 International Sym-
posium on Ubiquitous Virtual Reality (ISUVR), pages
1–3, Nara, Japan. IEEE.
Lopez, F., Navarro, D., and Sundstedt, V. (2017). Ethi-
cal Considerations for the Use of Virtual Reality: An
Evaluation of Practices in Academia and Industry. In
Lindeman, R. W., Bruder, G., and Iwai, D., editors,
ICAT-EGVE 2017 - International Conference on Arti-
ficial Reality and Telexistence and Eurographics Sym-
posium on Virtual Environments. The Eurographics
Association.
McMahan, R. P., Alon, A. J. D., Lazem, S., Beaton, R. J.,
Machaj, D., Schaefer, M., Silva, M. G., Leal, A.,
Hagan, R., and Bowman, D. A. (2010). Evaluat-
ing natural interaction techniques in video games. In
2010 IEEE Symposium on 3D User Interfaces (3DUI),
pages 11–14, Waltham, MA, USA. IEEE.
Peter Wozniak, Oliver Vauderwange, A. M. N. J. D. C.
(2016). Possible applications of the leap motion con-
troller for more interactive simulated experiments in
augmented or virtual reality.
Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D.
(2013). Analysis of the Accuracy and Robustness of
the Leap Motion Controller. Sensors, 13(5):6380–
6393.
HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications
110