Interaction Challenges in the Development of a Fire Warden VR

Training System using a HMD

Helen V. Diez

, Aitor Moreno

and Alejandro Garcia-Alonso

Vicomtech-IK4, San Sebastian, Spain

University of the Basque Country, San Sebastian, Spain

Keywords:

Virtual Reality, Head-Mounted Display, Interaction Devices.

Abstract:

This paper presents challenges encountered when interacting with 3D environments by means of Virtual Re-

ality devices in a seated position. A use case has been devised and developed to support this work: a ﬁre

warden virtual training system. The users interact with the virtual environment with a combination of a Head-

Mounted Display and other assisting devices to complement the virtual experience (keyboard, gamepad, leap

motion). This work points out the beneﬁts achieved using virtual reality in this environment and it exposes the

limitations experimented by humans in such situations. With the knowledge gathered with these experiments

this paper proposes interaction techniques to overcome them.

1 INTRODUCTION

In the last years, the new generation of Head-Mounted

Displays (HMDs) have become popular devices to

provide highly immersive experiences which are very

difﬁcult to achieve by other means. They are excellent

tools to view virtual environments (VEs) in a realistic

way but, by themselves, they do not offer interactive

capabilities with the objects in the VE. Additional de-

vices are attached to the HMD setup to overcome this

situation.

When wearing a HMD, users cannot see the real

world surrounding them, so it is difﬁcult to locate

and use the mouse, the keyboard, the gamepad or any

other interaction device at their disposal. Addition-

ally, even wireless devices, they are normally placed

at physical location, breaking the virtual experience

of the user. New interaction devices have been devel-

oped to solve this issue, such as, walking platforms,

speciﬁc hand input devices, haptic gloves, etc. These

solutions may be a good choice for gaming purposes,

in which a higher level of activity is necessary or rec-

ommendable. However, depending on the application

or ﬁnal user these devices may not be suitable. For

example, for educational purposes, with school stu-

dents, it seems more likely for them to use their own

computer screen and keyboard.

This work studies some interaction possibilities

of HMDs in a natural seated position. To analyse

the challenges of interacting with HMDs we con-

sider three universal interaction tasks within 3D or

VE applications (Hand, 1997), (Bowman et al., 2001),

(Bowman et al., 2004) which Jankowski, J. and Ha-

chet, M. (Jankowski and Hachet, 2013) deﬁned as:

• Navigation: Related to the motor task of moving

the viewpoint through an environment. If it in-

cludes a cognitive component, it is referred to as

Wayﬁnding

• Selection and Manipulation: Related to the tech-

niques of choosing and picking an object and

specifying its position, orientation, and scale.

• System Control: Related to the communication

between user and system which is not part of the

VE.

For the use case presented in this work, we will study

the methodology followed when planning the exper-

iments and the decisions taken regarding these three

tasks according to our work experience.

This paper is organized as follows. Section 2 anal-

yses the related work regarding interaction in VEs and

hand gesture recognition. Section 3 explains the in-

teraction devices selected for the Setup of this work.

Section 4 describes the ﬁre warden use case devel-

oped and analyses each of the interaction tasks. The

ﬁnal section is about conclusions and future work.

Diez H., Moreno A. and Garcia-Alonso A.

Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD.

DOI: 10.5220/0006133400840091

In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2017), pages 84-91

ISBN: 978-989-758-229-5

2 RELATED WORK

Jankowski, J. and Hachet, M. (Jankowski and Hachet,

2013) presented a state of the art of non-immersive

interaction techniques for Navigation, Selection and

Manipulation, and System Control. This work also

summarizes the main 3D Web design guidelines.

Moya, S. et al. (Moya et al., 2014) analysed the

inﬂuence of different factors in locomotion control in

3D VEs. They designed an automatic locomotion sys-

tem and concluded that, for non-casual gamers, auto-

matic locomotion is preferred, whereas gamers prefer

to control locomotion themselves.

Regarding hand gesture recognition for HCI,

(Sharma and Verma, 2015) proposed the tracking of

six static hand gestures. They used images extracted

from recorded video streams at different distances and

positions. They are able to use their system to control

applications such as power point, media player, win-

dows picture manager, etc.

Manresa, C. et al. (Manresa et al., 2005) present a

real-time algorithm to track and recognise hand ges-

tures for interacting with the videogame. This algo-

rithm is based on three main steps: hand segmenta-

tion, hand tracking and gesture recognition from hand

features. The results of this work show that users can

substitute traditional interaction metaphors with their

low-cost interface.

Segura, I. et al. (Segura et al., 2007) developed a

simulator of construction machinery for safety train-

ing. In this work the visualization was implemented

as a chroma-key-based mixed reality system, combin-

ing a 3D VE, a real cabin interior, and some superim-

posed messages to the user. The overall impressions

from the testers were positive.

Goenetxea, J. et al. (Goenetxea et al., 2010)

presented an interactive and stereoscopic hybrid 3D

viewer. In this work they used Nintendo Wii control

and developed a dynamic gesture recognition proce-

dure used for interacting with the weather animations.

Regarding ﬁre simulation, it has also been a re-

search topic in VR applications (Moreno et al., 2011),

(Moreno et al., 2012). The simulation of real-time in-

teractive ﬁre is a challenging topic that has not been

solved yet.

In this work we propose to use a HMD combined

with usual interaction devices such as keyboards and

gamepads. Leap Motion Controller is used for hand

recognition. The following sections explain the setup

we have developed for a ﬁre warden virtual training

use case.

3 USE CASE SETUP

In our use case, the tests are carried out autonomously

by a ﬁre warden trainee. Therefore, it is not a collabo-

rative learning experience, as each user must be aware

of their own actions and decisions.

The setup used in these experiments is composed

of the following items:

• Chair. One goal of our work is to keep the setup as

simple as possible. This is why we tried to limit

the amount of extra devices needed. The users

will interact in a seated position, needing no other

devices.

• Computer. The recommended system require-

ments are: Windows 7 or newer, Compatible

HDMI video output, Intel i5, NVIDIA GTX 770

or greater.

• Monitor. The ﬁnal goal is for the user to visual-

ize the 3D VE with the HMD, but it is necessary

to have a monitor to make the ﬁrst steps until the

virtual experience is launched and to get some ad-

ditional information after the training session.

• HMD. We have used Oculus Rift DK2. The Ocu-

lus CV version or any other HMD brand could be

used, but the speciﬁcations of the computer might

require modiﬁcations to match the HMD require-

ments.

• Keyboard. Anyone used to dealing with comput-

ers is familiarized to keyboards and there is no ex-

tra effort needed. Wireless or not is irrelevant, but

it should be placed in a known and ﬁxed place be-

fore the VR training session. Bare in mind that

the user will not be able to see the real world with

the HMD on.

• Gamepad. It is also a familiar device in video

games and it is intuitive for navigation purposes.

We have used the Logitech Rumblepad2.

• Leap Motion Controller. This device provides

means for tracking the hand gestures performed

by the user. It was attached to the Oculus Rift

with the universal bundle. The latest Orion beta

software has been used (Orion, 2016).

3.1 Interaction Devices

This section discusses the interaction devices selected

for the use case presented in this work.

3.1.1 Keyboard

The keyboard is used to navigate through the VE us-

ing the up, down, left and right arrow keys. In tra-

ditional computers, they are a group of isolated keys,

Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD

easily distinguishable from others. In more compact

devices such as laptops, the arrow keys despite being

closer to other keys are still easy to locate. The ad-

vantage with laptops is that the keyboard is attached

to the screen so it cannot change position. Sometimes,

WASD keys are also used for navigating. Normally,

this keys are used with the left hand leaving the right

hand free to interact with the mouse. This setup is

typical used in gaming (see Figure 1).

Both arrow keys and WASD keys can be used with

only three ﬁngers of one hand, leaving the other hand

free. In this use case, the free hand is used to perform

the gestures deﬁned to select and manipulate objects

in the scene. This gestures are captured by the Leap

Motion Controller.

3.1.2 Gamepad

Instead of using the keyboard to navigate, we have

experimented with a gamepad. In this case both hands

are needed to hold the gamepad and the user has to

let go of one hand to perform the hand gestures for

selecting and manipulating objects of the VE.

In most applications, the right stick is used for

navigating (forward, backward, left and right) and the

left stick for turning the camera view. This left stick

is not necessary as the user can look around by mov-

ing the head with the 360 tracking system offered by

the HMD. So, in the gamepad the user only has to use

the right stick, none of the other buttons are needed.

This makes it simple for the user, despite wearing the

HMD.

3.1.3 Leap Motion

HMDs offer a really immersive experience. Users

feel they are actually in the VE they are watching

through the HMD device. This experience is so re-

alistic and convincing that users try to interact with

objects from the virtual world with their own hands.

However, regular interaction devices as the ones de-

scribed above do not allow this sort of interaction.

This is why we have chosen Leap Motion Controller

for our work. This device allows users to integrate

their hands into the virtual world and unlike other de-

vices, they require no practice or previous knowledge

from the user.

The Leap Motion Controller is mounted on the

front of the Oculus Rift glasses, as seen in Figure 2,

and whenever the users lift their hands to grab an ob-

ject from the virtual world, the virtual hands appear in

the virtual world, performing exactly the same move-

ments as the real hands.

This setup is more compact than other alternatives

using external trackers for gesture recognition.

4 USE CASE: FIRE WARDEN

TRAINING

This use case has been designed to train experts on

occupational hazard prevention and more precisely in

ﬁre safety in buildings. The ﬁre warden trainee has to

perform the tasks described below:

• Locate the possible ﬁres in the building.

• Extinguish these ﬁres.

• Alert other people in the building.

• Evacuate the building.

These tasks imply different levels of interaction

with the VE. In the next subsections we will explain

the methodology followed to decide which kind of in-

teraction devices to use, and we will explain the trial

and error process carried out for each of the classical

VR interaction categories.

4.1 Navigation

One of the tasks that the user has to perform is the

location of the possible ﬁres throughout the building,

to discover these ﬁres the user has to navigate through

a two-ﬂoored warehouse. The ﬂoors are connected by

two staircases which the user can use as many times

as needed.

Another task that implies navigation is the evac-

uation of the building. For these tasks the user has

to know the exits available in the building. Moreover,

the users are asked to put the ﬁre out, to do that the lo-

cation of the extinguishers has to be found. The user

has to select them and head towards the ﬁre.

These tasks imply wayﬁnding, as the user has to

have a cognitive overview of the building and has to

be able to make decisions. To ease this job, the appli-

cation displays some messages (images or text) on the

screen of the HMD. These messages are explained in

section4.4.

4.2 Selection

One of the goals of our ﬁre safety training is to allow

wardens to learn how to use ﬁre extinguishers in a

safe and efﬁcient way. In real life a ﬁre extinguisher

must be manipulated in a precise way and ﬁres must

be put out following the PASS method (Safety and

Administration, 2016):

• Pull the pin in the handle.

• Aim the nozzle at the base of the ﬁre.

• Squeeze the lever slowly.

• Sweep from side to side.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

Figure 1: Arrow Keys and WASD Keys in a Traditional Keyboard (left) and a Laptop (right).

Figure 2: Leap Motion Controller attached to Oculus Rift

DK2.

Figure 3: Setup #1: Interaction with Keyboard and Leap

Motion Controller.

Figure 4: Setup #2: Interaction with Gamepad and Leap

Motion Controller.

We have made several tests to see which hand gestures

are most appropriate to simulate these actions in vir-

tual life and we have used the Leap Motion Controller

(LM) to track them.

In the ﬁrst place, the warden has to grab a ﬁre ex-

tinguisher. To allow this action in the ﬁre safety train-

ing application, ﬁrst the user has to navigate to one of

the extinguishers, and get close to it to a certain dis-

tance. We have set 60 cm as the maximum distance

to allow the user to grab the extinguisher. We have

chosen that distance as it is the standard length of an

adult’s arm (McDowell et al., 2008),(NASA, 2016).

Once the users are placed at reaching distance from

the extinguisher they have to extend their arm towards

it and grab it.

To represent this gesture, the ﬁrst idea we thought

of was a “palm closing” gesture. This gesture is easily

trackable by the LM and it is easy to perform and un-

derstand by anyone: “I want to grab the extinguisher”.

However, we dismissed this gesture for two main rea-

sons. First, the PASS method indicates the user has to

pin in the handle. This action implies a more precise

movement of the ﬁngers and we wanted to force the

user to do so. Second, as explained later, we selected

the “palm closing” gesture for opening and closing

doors or windows.

We want to enhance the interaction degree and the

learning experience. For that we designed indepen-

dent gestures for different actions. This way, the user

can easily internalize the meaning and consequences

of each virtual gesture in the real world. This is why

we decided to deﬁne the “pinch” gesture for grabbing

objects in the scene, in this case the ﬁre extinguisher

(see Figure 5).

So, summarizing the full action process, the user

has to get close to the extinguisher, reach an arm to-

wards it and pinch it by holding the index and thumb

ﬁngers together. The platform must detect a collision

between the extinguisher and the virtual representa-

tion of the user’s hand. If this happens, the extin-

guisher is moved towards the user and it is placed in

the right side of the visual ﬁeld, representing that the

extinguisher has been grabbed and it is ready to be

used. The actions that the user can do with the extin-

guisher are discussed in the next subsection 4.3.

Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD

Figure 5: “Pinch” gesture with Setup #1 (left) and Setup #2 (right).

Another goal, is to teach the warden how to evac-

uate the building safely. In this work, to do this, the

warden has to navigate to the closest exit and open the

door. We have chosen the “palm closing” gesture for

this action as it is the one we perform in real life to

grab door handles. The user has to approach a door

in the virtual scene and reach out the arm towards it.

As with the extinguisher selection, the user has to be

placed at least 60 cm. from the door.

At ﬁrst, we thought on detecting the collision be-

tween the virtual hand and the door handle. We also

thought it would be recommendable to track the hand

turning down a certain angle as if turning a door han-

dle to open it. But these actions resulted confusing

and tedious. In the virtual scene, to detect a collision

between the hand and the handle, the warden had to

get very close to the door, which implied correcting

their position and performing the “palm closing” ges-

ture several times. Moreover, depending on the an-

gle degree the hand was turned, this gesture could be

confused with the “thumbs up” gesture. Finally, we

decided to simplify this action by allowing the door

to open by performing the “palm closing” gesture at

the right distance from an exit (see Figure 6).

Additionally, we have the same action to represent

a door by pulling it or by pushing it. Even if pulling

or pushing would affect in the cognitive process of

opening a door, we decided to omit this feature for the

sake of simplicity. Furthermore, it is expected that the

doors should be correctly designed, promoting “push-

ing” mode over “pulling”, as it may be required for

safety regulations.

4.3 Manipulation

In this use case, the ﬁre warden must learn to manip-

ulate ﬁre extinguishers. First, the user has to navigate

with the ﬁre extinguisher towards the ﬁre and manip-

ulate it following the PASS method. The correct dis-

tance to put out a ﬁre safely is between three to one

meters. So if the user is further than 3 meters or closer

than 1 meter from the ﬁre an alert text is displayed as

a HUD. Once the user is at the right distance, the user

has to aim at the ﬁre, squeeze the nozzle and sweep

from side to side.

In real life, we would need both hands to manip-

ulate the extinguisher. However, in this virtual sce-

nario the user also has to interact with the keyboard

or gamepad. When using the keyboard, it is not con-

venient for the user to lift the ﬁngers from the arrow

keys and with the gamepad, the user needs one hand

to keep holding it. To simulate the squeezing of the

nozzle, we decided to track one hand of the user per-

forming a “thumbs up” gesture (Figure 7), closing the

hand with the thumb heading up. We chose this ges-

ture because it is similar to the one users should per-

form in real life and to differentiate it from the “clos-

ing palm” gesture, in which the thumb is not visible.

Then, if the user is in possession of an extin-

guisher and performs the “thumbs up” gesture at the

right distance from a ﬁre, a graphical particle system

representing the extinguishing jet will blast out from

the user’s hand position. If the user stops performing

the gesture, the particle jet stops. To increase the real-

ism, the noise of a real extinguisher is played through

the audio systems (integrated in the HMD or exter-

nal).

While performing this gesture the user also has to

move the hand from left to right to perform a sweep-

ing gesture.

Our Unity-based 3D engine (Unity3D, 2016) de-

tects whether a collision between the particle jet and

the ﬁre has taken place. This way we check if the

user is aiming at the ﬁre and sweeping the hand cor-

rectly. If there is a collision the ﬁre is put out. We

deﬁne each ﬁre as a collection of particle systems, so

they are put out gradually. If the user does not sweep

from side to side, the ﬁre will not put out correctly.

Fire extinguishers contain around 10 seconds of ex-

tinguishing power. If the warden spends more than

that time trying to extinguish the ﬁre it will probably

mean the sweeping is not being performed correctly.

In this case an alert HUD appears to remind the war-

den the PASS method.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

Figure 6: “Palm closing” gesture with Setup #1 (left) and Setup #2 (right).

Figure 7: “Thumbs up” gesture with Setup #1 (left) and Setup #2 (right).

Figure 8: Mode Selection.

4.4 System Control

This section explains the input messages we have de-

ﬁned to communicate the user and the system. These

messages appear as HUDs (Heads-Up displays) on

the screen.

Where to place each HUD is also important, as

far as possible they should be integrated in the virtual

scene (Yao et al., 2014).

The user can begin the simulation in training mode

or in simulation mode (see Figure 8). The user selects

between these options with the LM.

During the simulation the user can visualize

a semi-transparent and non-intrusive 2D mini-map.

This map has been designed as an egocentric map.

The map shows the following signs to help the user:

• “You are here” marker.

Figure 9: Upper Left: 2D mini-map. Upper Right: Timer.

• Exit location signs.

• Extinguisher location signs.

A timer is also visible at all times, so the user knows

how long it is taking to perform the exercise. These

HUDs are shown in Figure 9.

In the training mode, some green arrows are dis-

played on the 3D environment to show the user the

path to the closest extinguisher, once the user has se-

lected the extinguisher and put out the ﬁre, again some

arrows are displayed to show the path to the clos-

est exit (see Figure 10). These signs are not shown

in the simulation mode, as the user should have ac-

quired knowledge about the environment in the train-

ing mode.

Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD

Figure 10: Training Mode. Green arrows placed on the ﬂoor

guide the user to the exit.

5 CONCLUSIONS

In this work, a VR ﬁre warden training system has

been presented. Its main objective is to provide VR

experiences to ﬁre wardens in order to internalize

and comprehend the concepts behind the standard-

ized procedures that have to be followed when a ﬁre

emergency arises in a building. In our preliminary

developments, a few actions have been tested as they

target the main interactions within VR environments:

Navigation for evacuation routes and ﬁre ﬁnding; Se-

lection and Manipulation to interact with the extin-

guishers and doors; and System Control interaction

to display non intrusive 2D information as a HUD.

All the actions have been developed as a combination

of gestures recognized with the aid of Leap Motion

Controller and keyboard/gamepad to navigate the 3D

environment.

The seated position has been deﬁned as a require-

ment from the very beginning as it is the most com-

fortable way of interacting with the VR environment.

The stand-up position poses additional problems like

i) problems with the cabling, ii) collisions with the

real-world furniture and iii) anxiety regarding the

cognitive disassociation of the virtual position and the

real position, specially when stairs are included in the

scenarios. The seated position is also recommendable

if the interaction is going to last a long period of time.

More trials and validation of the VR setup is

needed. However, we have found that the election of

the keyboard or the gamepad is a matter of personal

preferences. In any case, we have reports about the

necessity to adapt the keys of the keyboard or buttons

of the gamepad to each personal preference.

Some users found some limitations in the VR nav-

igation. They report that turning commands conﬂict

with looking to the sides in the VR environment. Nev-

ertheless, it is a matter of time to get the users ac-

quainted with the navigation system.

As future work, the following prototype will ex-

periment alternatives to solve the problems reported

by the users. A more extensive evaluation will be car-

ried out.

The extension of the prototype to other users and

purposes is also under consideration. The presented

use case could be adapted easily to children in order

to show them basic information about what they have

to do when a ﬁre emergency arises.

After the preliminary implementation, we are

planning to introduce a different use case, oriented to

the visualization of 3D models on the Web. This use

case is very different from the one presented in this

work and therefore, it will provide complimentary in-

formation about the VR techniques in a different VR

environment.

REFERENCES

Bowman, D., Kruijff, E., LaViola Jr, J. J., and Poupyrev,

I. (2004). 3D User Interfaces: Theory and Practice,

CourseSmart eTextbook. Addison-Wesley.

Bowman, D. A., Kruijff, E., LaViola Jr, J. J., and Poupyrev,

I. (2001). An introduction to 3-d user interface de-

sign. Presence: Teleoperators and virtual environ-

ments, 10(1):96–108.

Goenetxea, J., Moreno, A., Unzueta, L., Galds, A., and Se-

gura, l. (2010). Interactive and stereoscopic hybrid

3d viewer of radar data with gesture recognition. In

Graa Romay, M., Corchado, E., and Garcia Sebas-

tian, M., editors, Hybrid Artiﬁcial Intelligence Sys-

tems, volume 6076 of Lecture Notes in Computer Sci-

ence, pages 213–220. Springer Berlin / Heidelberg.

Hand, C. (1997). A survey of 3d interaction techniques.

In Computer graphics forum, volume 16, pages 269–

281. Wiley Online Library.

Jankowski, J. and Hachet, M. (2013). A survey of inter-

action techniques for interactive 3d environments. In

Eurographics 2013-STAR.

Manresa, C., Varona, J., Mas, R., and Perales, F. (2005).

Hand tracking and gesture recognition for human-

computer interaction. ELCVIA Electronic Letters on

Computer Vision and Image Analysis, 5(3):96–104.

McDowell, M. A., Fryar, C. D., Ogden, C. L., and Flegal,

K. M. (2008). Anthropometric reference data for chil-

dren and adults: United states, 2003–2006. National

health statistics reports, 10(1-45):5.

Moreno, A., Segura,

A., Korchi, A., Posada, J., and Otaegui,

O. (2011). Interactive urban and forest ﬁre simulation

with extinguishment support. In Advances in 3D Geo-

Information Sciences, pages 131–148. Springer Berlin

Heidelberg.

Moreno, A., Segura,

A., Zlatanova, S., Posada, J., and

Garc

ıa-Alonso, A. (2012). Beneﬁt of the integration

of semantic 3d models in a ﬁre-ﬁghting vr simulator.

Applied Geomatics, 4(3):143–153.

Moya, S., Grau, S., and Tost, D. (2014). First-person loco-

motion in 3d virtual environments: a usability analy-

sis. J. UCS, 20(7):1026–1045.

HUCAPP 2017 - International Conference on Human Computer Interaction Theory and Applications

NASA (2016). Antrhropometry and biomechanics.

Orion (2016). Orion beta sw.

Safety, T. O. and Administration, H. (2016). Fire protection

and prevention.

Segura, l., Moreno, A., Brunetti, G., and Henn, T. (2007).

Interaction and ergonomics issues in the development

of a mixed reality construction machinery simulator

for safety training. In Dainoff, M., editor, Ergonomics

and Health Aspects of Work with Computers, volume

4566 of Lecture Notes in Computer Science, pages

290–299. Springer Berlin / Heidelberg.

Sharma, R. P. and Verma, G. K. (2015). Human computer

interaction using hand gesture. Procedia Computer

Science, 54:721 – 727.

Unity3D (2016). Unity3d.

Yao, R., Heath, T., Davies, A., Forsyth, T., Mitchell, N., and

Hoberman, P. (2014). Oculus vr best practices guide.

Oculus VR.

Interaction Challenges in the Development of a Fire Warden VR Training System using a HMD