HandWindowTeleportation: Locomotion with Hand Gestures for Virtual

Reality Games

Hibiki Kirihata

and Tomokazu Ishikawa

1,2 a

Toyo University, 1-7-11 Akabanedai Kita-ku, Tokyo, Japan

Prometech CG Research, 3-34-3 Hongo bunkyo-ku, Tokyo, Japan

Keywords:

Virtual Reality, Point and Teleport, Hand Tracking, Hand Gesture.

Abstract:

This study designs and evaluates a novel one-handed gesture-based control method for teleportation within a

VR game space. For three different stages, we measured travel time, number of moving maneuvers, and ac-

curacy in achieving checkpoints, and used two questionnaires, NASA-TLX and SUS, to evaluate the subject’s

workload and the usability of proposed system. It was suggested that the proposed teleportation method using

hand gestures is appropriate for games that require agility, such as action games, and is less tiring and has

better ease of use than the previous method and a method on existing products.

1 INTRODUCTION

Recent technological developments have made virtual

reality (VR) head-mounted displays (HMDs) more

widespread and affordable. An increasing number

of HMDs utilize the inside-out method, do not use

a controller, and are equipped with hand-tracking

functions. With plans to release HMDs without

controllers, hand tracking will become an important

method of operation in the XR (Extended Reality)

environment in the future. Indeed, there is a lot of

research on manipulating virtual objects using hand

tracking (Sch

afer et al., 2022; Hameed et al., 2021;

Pei et al., 2022).

Because many VR applications and games involve

frequent movement, optimizing this behavior is an

important research topic. VR experiences using VR

HDMs would be more highly immersive if they were

operated by actually walking, but this is often difﬁ-

cult due to physical limitations. Therefore, a method

to move around in VR space without having to walk

in real space: Point And Teleport was proposed. Con-

ventional VR HMDs always have a controller and are

assumed to be operated via a controller. As mentioned

previously, in light of the information on future HMD

releases, a movement method based on hand tracking

without a controller is needed. There are few research

examples of movement methods using hand tracking,

and to the best of our knowledge, there are few exper-

https://orcid.org/0000-0002-9176-1336

iments comparing the improvement of VR experience

by different movement methods. In particular, there

are almost no examples of testing whether it is easy

to move while performing actions other than moving,

such as attacking an enemy, that are intended to be

used in actual games. Therefore, in this study, assum-

ing that it will be used in a game, we implement a

moving method that is quicker and more intuitive to

use, and compare our method to a previous method

and a method used in an existing product.

2 RELATED WORKS

In this section, we describe previous researches, clas-

sifying them into two categories: moving methods in

VR space and moving methods using hand tracking.

2.1 Moving Methods in VR Space

Bolte et al. proposed a method called “jumper

metaphor,” which supports actual walking for short

distances and virtual jumping for longer dis-

tances (Bolte et al., 2011). According to evalua-

tions, this method allows for more effective explo-

ration compared to actual walking and has little effect

on spatial perception.

The research by Bozgeyikli et al. presented a

new locomotion technology called ”point and tele-

port” and compared it to the gait simulation and joy-

170

Kirihata, H. and Ishikawa, T.

HandWindowTeleportation: Locomotion with Hand Gestures for Virtual Reality Games.

DOI: 10.5220/0012472900003660

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2024) - Volume 1: GRAPP, HUCAPP

and IVAPP, pages 170-176

ISBN: 978-989-758-679-8; ISSN: 2184-4321

sticks that are normally used (Bozgeyikli et al., 2016).

With this technology, the user simply points to where

they want to go in the virtual world and is teleported

to that location. One major advantage is that there

is no visible parallel movement, which is expected to

reduce the likelihood of motion sickness. 16 users

participated in the experiment, suggesting that while

”point and teleport” is a fun and user-friendly locomo-

tion technique, the additional directional component

degrades the user experience.

Funk et al. proposed and evaluated three dif-

ferent point-and-teleport techniques that allow the

user to specify the target orientation during telepor-

tation (Funk et al., 2019). The results showed that the

teleportation technique with directional instructions

increased the average teleportation time but decreased

the need to correct the orientation after teleportation.

Rantala and Kangas highlighted that teleportation

in VR space may limit spatial awareness because the

method of movement is not continuous (Rantala et al.,

2021). Teleportation is well suited for VR controllers

and can minimize simulatorsickness (motion sick-

ness), but may reduce spatial awareness compared

to continuous movement. The goal of their study

was to develop a continuous, controller-based move-

ment technique and to test whether it supports spa-

tial awareness. Rantala and Kangas introduced two

new techniques, “slider” and “grab,” and compared

them to teleportation. Results showed that “slider”

and “grab” were signiﬁcantly faster than teleporta-

tion, and that they caused signiﬁcantly less simulator

sickness than teleportation. Furthermore, it was con-

cluded that the continuous technique provided better

spatial awareness than teleportation.

Cmentowski et al. propose a novel augmented

walking approach for virtual reality games that

presents a virtual tunnel covering the entire travel dis-

tance (Cmentowski et al., 2022). This virtual tunnel

hides the visual ﬂow from the applied motion accel-

eration, while the actual accelerated motion is visible

through the tunnel wall windows. According to the

evaluation, this approach avoids cybersickness and

at the same time improves physical activity and pre-

serves presence.

Considering that additional cognitive load could

negatively affect performance for the method pro-

posed by Funk et al. (Funk et al., 2019), Mori et

al. proposed a new P&T design: “points to teleport”

(P2T) (Mori et al., 2023). They also reevaluated the

accuracy of P&T with the user standing and sitting.

In these studies, researchers are experimenting

with operability by designing a UI that mainly uses a

VR controller when moving and presents the telepor-

tation destination on a virtual environment. In view of

the future development of VR devices, we will inves-

tigate a method of teleportation using hand gestures

without a controller.

2.2 Moving Methods Using Hand

Tracking

Focusing on a technique to control teleport-type

movement with hand gestures, Sch

afer et al. pro-

posed a way for users to move in VR using only

their hands; two both-handed and two one-handed

techniques were evaluated in 21 participants accord-

ing to effectiveness and efﬁciency and user prefer-

ence (Sch

afer et al., 2021). When performing hand

gestures, there was no clear difference between using

both hands or only single hand, so they concluded that

one hand alone can comfortably and effectively move

through the virtual world.

The study by Neamoniti and Kasapakis presents

preliminary evaluation results on the use of motion

tracking versus hand tracking in IVR (Immersive

Virtual Reality) games (Neamoniti and Kasapakis,

2022). Results indicate that while hand tracking has

lower levels of ease of use and learning effectiveness,

it does not affect the overall game experience as com-

pared to motion controllers.

Lesaca et al. conducted an experiment comparing

four teleportation methods for everyday movement in

VR (Lesaca et al., 2022). The results of the experi-

ment show that in general use, experienced VR users

prefer to use their hands to control the virtual arc and

indicate the location and direction they wish to tele-

port. However, Lesaca et al. concluded that the step-

by-step teleportation method in the direction of gaze

supports more natural movement and may encourage

shorter travel paths, but instead takes longer travel

times.

With reference to the above related works, we

design and evaluate a hand gesture-based moving

method that can be used in games, which is better than

the previous methods.

3 PROPOSED METHOD AND

EXPERIMENT

The hand gesture of the moving method proposed in

this research is designed with an emphasis on the abil-

ity to move instantly, at the user’s will, to the co-

ordinates targeted by the user. We designed hand

gestures inspired by ”The Grab-a-Scope” is one of

gadgets which Doraemon has. The index ﬁnger and

thumb which are spread out liken to a scope, and the

HandWindowTeleportation: Locomotion with Hand Gestures for Virtual Reality Games

171

Figure 1: Image sequences when a hand gesture for each moving method is performed. Top: Oculus Integration, middle:

Two-Hand Palm and bottom: “HandWindow” (ours).

Figure 2: Image sequences when hand gestures other than movement are performed. Top: turning, middle: normal attack and

bottom: charge attack.

user pinch such as plucks out the landscape, which

results in moving to its coordinate. We call this tele-

portation method using hand gesture “HandWindow”.

Our method employs a user interface that speciﬁes

the teleportation destination as a straight line from the

camera coordinates to the ground surface connecting

the midpoints of the thumb and index ﬁnger. We con-

sidered that a straight guideline would make it eas-

ier for the user to aim at the destination compared to

the previous method. By touching the index ﬁnger

and thumb, the user decides to move. Using this hand

gesture, we expect the user to have the experience of

GRAPP 2024 - 19th International Conference on Computer Graphics Theory and Applications

172

picking the place at hand.

We implement a system to identify hand gestures

using hand tracking running on the Meta Quest series.

We employ the framework provided by the Oculus In-

tegration to recognize the user’s hand. The angle of

the ﬁnger joints of the hand on the virtual, which is

recognized by Oculus Integration, is acquired to de-

termine how bent the ﬁngers are at that angle. Us-

ing the above methods, we implement three differ-

ent moving methods (Oculus Integration, a previous

method and “HandWindow”), a turning method, and

two attack methods on a VR game. In Oculus Integra-

tion, an arc-shaped guideline is drawn from between

the index ﬁnger and thumb to select the teleportation

destination. The decision is made by touching the in-

dex ﬁnger and thumb, and the user teleport there. As

a previous method for comparison, we implemented

Two-Hand Palm which is the method proposed by

Sch

afer et al. (Sch

afer et al., 2021). In the Two-Hand

Palm, an arc-shaped guideline is output from the palm

of the hand to select the teleportation destination. The

user executes teleportation by bending the index ﬁn-

ger of the other hand. For each moving gesture, image

sequences are shown in Figure 1.

To conduct the experiment in combination with an

action game, hand gestures are assigned to actions re-

quired in addition to the moving method. Turning is a

directional change in the player’s direction; by sweep-

ing the space below the eye with the palm, the player

rotates in the direction swept by the palm. The two

types of attacks are the normal attack, which involves

waving the hand with the index and middle ﬁngers

held up, and the charge attack, which involves form-

ing a bowl with both hands and ﬁring an energy shot.

Normal attacks can be ﬁred in rapid succession but are

less powerful and difﬁcult to gesture. Charge attacks

take longer to strike but are more powerful and can

be used with a simple gesture. It is designed so that

those who can do a normal attack will use it as their

main tool, and those who have difﬁculty with it will

use a charge attack. Images sequences of the turning

and attacking gestures are shown in Figure 2.

Experimental participants were asked to use three

different moving methods on the implemented origi-

nal VR game, and their movement times and move-

ment coordinates were recorded. The stages for the

moving experiment were prepared in three patterns: a

ﬂat and straight stage, a narrow and bent stage, and

a stage with a difference in height. One stage for

the combat experiment was prepared. In the combat

stage, the player is limited to moving range. The ap-

pearance of the experimental stage is shown in Fig-

ure 3 and Figure 4 shows the combat stage. Users

are asked to play in two modes: one in which loca-

(a) Stage 1: a ﬂat and straight stage

(b) Stage 2: a narrow and bent stage

Figure 3: Appearance of each stage. A red marker indicates

the start point, a blue marker indicates the goal point, and

yellows marker indicate checkpoints that the user must pass

through.

Figure 4: Appearance of stage 4. A red marker indicates the

start point, and blacks marker indicate the locations where

enemies appear. The black wall prevents the player from

going to the location where the enemy appears.

tions to be passed are indicated as checkpoints, and

the other in which they are free to reach the goal. That

is, we ask one user to play 21 different experimen-

tal patterns. From this measurement data, the total

travel time from start to goal, the number of moving

operations, and the accuracy of reaching checkpoints

HandWindowTeleportation: Locomotion with Hand Gestures for Virtual Reality Games

173

Figure 5: During the experiment. The subject sits in a chair

and plays a game in VR, using only hand gestures to rotate

and move without changing the position or direction of the

body.

are calculated and used for evaluation. After using

the moving methods, the respondents were asked to

evaluate each of the moving methods in a question-

naire that included NASA-TLX (NASA Task Load

Index) (Hart and Staveland, 1988) and SUS (System

Usability Scale) (Brooke, 1995). Participants also an-

swer to respond to the best method of travel used.

We use a PC with OS: Windows 10 Home 64bit,

CPU: Intel® Core™ i7-10700K CPU @ 3.80GHz,

RAM: 16GB and GPU : NVIDIA GeForce RTX 2080

SUPER 8GB and Meta Quest 3 for our experiments.

We develop a game for experiment on Unity.

4 RESULTS AND DISCUSSION

11 male university students in their teens and twenties

majoring in information science were asked to coop-

erate in the experiment. Only two of them had never

experienced VR. The order of movement methods in

the experiment was randomized to avoid differences

in training level. The Figure 5 shows the situation

during the experiment.

First, the results for total travel time and num-

ber of movements are shown in the Figure 6 and 7

for the experiment in which travel was performed re-

gardless of checkpoints. Averaging the arrival time

of all stages, the Oculus Integration, Two-Handed

Palm, and the proposed method took 15.28 s, 15.38

s, and 11.60 s, respectively. We found the possibil-

ity of reducing travel time for any stage compared

to the existing method. The proposed method has a

small variance for all stages, is equally easy to use

for all users, and takes less time in tight and winding

places.Therefore, we can mention that our method is a

method with a small turnaround. However, using the

Figure 6: Box plots of total travel time in experiments in

which travel is independent of checkpoints.

Figure 7: Box plots of number of moving method in exper-

iments in which travel is independent of checkpoints.

Figure 8: Accuracy comparison of move coordinates

against checkpoint coordinates.

proposed method took longer than the other methods

for the stages with height differences. We consider

that this result is caused by the fact that only “Hand-

Window” speciﬁes the teleportation destination as a

straight line, not as an arc. If the destination is spec-

GRAPP 2024 - 19th International Conference on Computer Graphics Theory and Applications

174

Figure 9: Results of counting the number of times a person

is hit in battle.

Figure 10: Box plots of NASA-TLX scores.

iﬁed as an arc, as in the previous method, it is possi-

ble to teleport to a location where the player cannot

see the ground. However, the number of movements

tended to be less than for other methods.

Next, the accuracy results are shown in Figure 8

for an experiment in which the participants are asked

to move through the checkpoints sequentially. The

RMSE (Root Mean Squared Error) was used to eval-

uate the degree of error in the transition destination by

the moving method with respect to the target check-

point. We conclude that the proposed method is

highly accurate because the variance is small and the

mean is also small.

The number of times a player was hit by bullets in

a battle was counted as a way to check if the method is

suitable for movement in an action game. The method

is easier to avoid if the number of being bombed times

is small. As you can be seen in the Figure 9, the

proposed method reduces the number of being shot.

Thus we can conclude that it is highly compatible

with games that require agility, such as action games.

The NASA-TLX scores for each moving method

Figure 11: Box plots of SUS scores. The blue broken line

shows the average SUS value of 68.

Figure 12: Results of asking experimental participants to

vote for the best 1 of the 3 moving methods in terms of ”ease

of moving”, ”ease of combat”, and ”overall evaluation”.

are shown in Figure 10. The higher the NASA-TLX

score, the higher the subject’s fatigue level. The box

plots show that the proposed method is less fatigu-

ing than the other two methods. The mean NASA-

TLX scores were 43.82, 37.76, and 27.94 for the Ocu-

lus Integration, Two-Handed Palm, and the proposed

method, respectively.

The results of the SUS scores are shown in the

Figure 11; the higher the SUS score, the better the

usability. A score above 68, the average of SUS,

is considered good usability, and a score above 80

is considered excellent usability. The SUS scores

were 55.91, 55.00, and 75.45 for the Oculus Integra-

tion, Two-Handed Palm, and “HandWindow”, respec-

tively. The proposed method has a better SUS score

than the other methods, with the average being close

to 80, and can be said to have excellent usability.

Finally, the Figure 12 shows the results of hav-

ing the participants vote for the best 1 among the 3

movement methods in terms of “ease of movement,”

HandWindowTeleportation: Locomotion with Hand Gestures for Virtual Reality Games

175

“ease of combat,” and “overall evaluation”. The re-

sults showed that the proposed method was the best

for all items. This result is considered to be voted

by the subjects as an overall evaluation of the above-

mentioned factors.

5 CONCLUSIONS AND FUTURE

WORK

In this research, we proposed a new single handed

gesture method for instantaneous movement within

VR space. The hand gesture implemented as the

proposed method was designed for action games, re-

specting the user’s initiative and allowing for agile

movement. In order to compare the proposed method

with the existing methods and the methods used in ex-

isting products, users were tested on three stages for

teleportation. Quantitative evaluations such as time

from start to ﬁnish, number of actions, accuracy on

each checkpoint NASA-TLX and SUS, as well as

subjective evaluations were conducted. For all the

items, the proposed method obtained positive results,

and we consider that the small turnaround in particu-

lar makes the proposed method more friendly to ac-

tion games.

We would like to discuss some improvements to

the proposed “HandWindow”. First, the problem that

only the ﬂoor can be speciﬁed as the destination,

making it difﬁcult to move depending on the terrain,

should be resolved. We think it is necessary to take

countermeasures such as making a downward judg-

ment when it hits a wall, or performing a sphere cast

when a ray cast is performed and movement is not

possible. There was an opinion that the UI should

display not only the destination but also a line from

the hand to the destination, which will be improved.

ACKNOWLEDGEMENTS

This work was supported by Toyo University Top Pri-

ority Research Program.

REFERENCES

Bolte, B., Steinicke, F., and Bruder, G. (2011). The jumper

metaphor: an effective navigation technique for im-

mersive display setups. Proceedings of virtual reality

international conference, 1(2).

Bozgeyikli, E., Raij, A., Katkoori, S., and Dubey, R. (2016).

Point & teleport locomotion technique for virtual re-

ality. In Proceedings of the 2016 Annual Symposium

on Computer-Human Interaction in Play, CHI PLAY

’16, page 205–216, New York, NY, USA. Association

for Computing Machinery.

Brooke, J. (1995). SUS: A quick and dirty usability scale.

Usability Eval. Ind., 189.

Cmentowski, S., Kievelitz, F., and Krueger, J. H. (2022).

Outpace reality: A novel augmented-walking tech-

nique for virtual reality games. Proc. ACM Hum.-

Comput. Interact., 6(CHI PLAY).

Funk, M., M

uller, F., Fendrich, M., Shene, M., Kolvenbach,

M., Dobbertin, N., G

unther, S., and M

uhlh

auser, M.

(2019). Assessing the accuracy of point & teleport lo-

comotion with orientation indication for virtual reality

using curved trajectories. In Proceedings of the 2019

CHI Conference on Human Factors in Computing Sys-

tems, CHI ’19, page 1–12, New York, NY, USA. As-

sociation for Computing Machinery.

Hameed, A., Perkis, A., and M

oller, S. (2021). Evaluating

hand-tracking interaction for performing motor-tasks

in VR learning environments. In 2021 13th Interna-

tional Conference on Quality of Multimedia Experi-

ence (QoMEX), pages 219–224.

Hart, S. G. and Staveland, L. E. (1988). Development

of NASA-TLX (task load index): Results of empiri-

cal and theoretical research. In Hancock, P. A. and

Meshkati, N., editors, Human Mental Workload, vol-

ume 52 of Advances in Psychology, pages 139–183.

North-Holland.

Lesaca, D., Cheung, H., Jena, T., and Cliburn, D. (2022).

Comparing teleportation methods for travel in every-

day virtual reality. In 2022 IEEE Conference on Vir-

tual Reality and 3D User Interfaces Abstracts and

Workshops (VRW), pages 238–242.

Mori, S., Hashiguchi, S., Shibata, F., and Kimura, A.

(2023). Point & teleport with orientation speciﬁcation,

revisited: Is natural turning always superior? Journal

of Information Processing, 31:392–403.

Neamoniti, S. and Kasapakis, V. (2022). Hand tracking vs

motion controllers: The effects on immersive virtual

reality game experience. In 2022 IEEE International

Symposium on Multimedia (ISM), pages 206–207.

Pei, S., Chen, A., Lee, J., and Zhang, Y. (2022). Hand in-

terfaces: Using hands to imitate objects in AR/VR for

expressive interactions. In Proceedings of the 2022

CHI Conference on Human Factors in Computing Sys-

tems, CHI ’22, New York, NY, USA. Association for

Computing Machinery.

Rantala, J., Kangas, J., Koskinen, O., Nukarinen, T., and

Raisamo, R. (2021). Comparison of controller-based

locomotion techniques for visual observation in vir-

tual reality. Multimodal Technologies and Interaction,

5:31.

Sch

afer, A., Reis, G., and Stricker, D. (2021). Control-

ling teleportation-based locomotion in virtual real-

ity with hand gestures: A comparative evaluation of

two-handed and one-handed techniques. Electronics,

10(6).

Sch

afer, A., Reis, G., and Stricker, D. (2022). Comparing

controller with the hand gestures pinch and grab for

picking up and placing virtual objects. In 2022 IEEE

Conference on Virtual Reality and 3D User Interfaces

Abstracts and Workshops (VRW), pages 738–739.

GRAPP 2024 - 19th International Conference on Computer Graphics Theory and Applications

176