A Physiological Evaluation of Immersive Experience of a View Control

Method using Eyelid EMG

Masaki Omata

, Satoshi Kagoshima

and Yasunari Suzuki

Interdisciplinary Graduate School of Medicine and Engineering, University of Yamanashi, Takeda, Kofu, Japan

Department of Computer Science and Media Engineering, University of Yamanashi, Takeda, Kofu, Japan

Keywords:

physiological Evaluation, Immersive Impression, Eyelid-Movement, Electromyogram, Virtual Environment.

Abstract:

This paper describes that the number of blood-volume pulses (BVP) and the level of skin conductance (SC)

increased more with increasing immersive impression with a view control method using eyelid electromyo-

graphy in virtual environment (VE) than those with a mouse control method. We have developed the view

control method and the visual feedback associated with electromyography (EMG) signals of movements of

user’s eyelids. The method provides a user with more immersive experiences in a virtual environment because

of strong relationship between eyelid movement and visual feedback. This paper reports a physiological evalu-

ation experiment to compare it with a common mouse input method by measuring subjects’ physiological data

of their fear of an open high place in a virtual environment. Based on the results, we ﬁnd the eyelid-movement

input method improves the user’s immersive impression more signiﬁcantly than the mouse input method.

1 INTRODUCTION

In recent years, some devices have been developed on

how to provide a user with more immersive impres-

sion in a virtual environment (VE)(Nagahara et al.,

2005), (Meehan et al., 2002). In the research ﬁeld,

some visual displays have been developed as visually-

immersive devices such as a head-mounted display

that covers the user’s view and peripheral vision, a

wall-sized screen that covers the user’s whole body

and 3D display that give a user a sense of depth. On

the other hand, some interaction techniques in a vir-

tual environment have been designed (Touyama et al.,

2006), (Ries et al., 2008), (Asai et al., 2002), (Man-

ders et al., 2008), (Steinicke et al., 2009), (Miyashita

et al., 2008), (Haffegee et al., 2007), (Kikuya et al.,

2011). Particularly, it is important to provide a user

with embodiment interaction techniques in a virtual

environment, just like the same as natural embodi-

ment motion in the real world. Therefore, several ges-

tural interaction techniques and several walk-through

environments have been developed. By using the in-

teraction techniques and the displays, users can more

naturally interact with a virtual environment in where

visual feedback to the users corresponds with motor

skills of the users.

However, these existing embodied interaction

techniques have some shortcomings of the correspon-

dence between motor skills and visual feedback. For

instance, people usually scrunch up their eyes to look

carefully at an important point. Conversely, people

usually open their eyes widely with to get a wide view

of surrounding area. Existing virtual reality systems

do not incorporate the basic movement like looking.

They usually provide users with control devices such

as a mouse, a joystick and a location sensor to control

user’s view. Because there is no real-world-like rela-

tionship (eye movement and view) between the device

operations and the visual feedback, the device opera-

tions may make the user aware of the gap between

the real world in where the user controls the device

and the virtual world in where the user sees and may

reduce the user’s immersive impression.

We, therefore, have developed a view control

method and the visual feedback associated with elec-

tromyographic (EMG) signals of movements of user’s

eyelids (TheAuthors, 2011). The method provides

users with more immersive experiences in a virtual

environment because of the strong relationships be-

tween the eyelid movements and the visual feedback.

Moreover, the visual feedback enhances a user’s vi-

sual functions in a virtual world, such as zoom-in/out

and see-through, by following the relationship be-

tween motor skills of seeing and views in the real

world. We think that the method will be used for

a natural and direct view control system in a three-

224

Omata M., Kagoshima S. and Suzuki Y..

A Physiological Evaluation of Immersive Experience of a View Control Method using Eyelid EMG.

DOI: 10.5220/0004719102240231

In Proceedings of the International Conference on Physiological Computing Systems (PhyCS-2014), pages 224-231

ISBN: 978-989-758-006-2

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

dimensional virtual environment.

Another research issue of VE interaction is how

to evaluate user’s immersive experience. A question-

naire, such as Likert scale or semantic differential

method, is one of conventional evaluation techniques,

but it is not continuous and not objective. Addition-

ally, it is difﬁcult to measure and evaluate sensation

by conducting a performance evaluation such as task

completion time and error ratio. We therefore con-

ducted a physiological evaluation test to compare the

immersive impression with the eyelid-movement in-

put method with that with a mouse input method.

This paper describes related work, the develop-

ment summary of the system, the physiological eval-

uation test about the users’ immersive impression and

improvement of the recognition algorithm of EMG of

user’s eyelid movements.

2 RELATED WORK

Asai et al. developed a vision-based interface based

on body position for viewpoint control in an immer-

sive projection display by tracking 3D positions of

the arms and head of the user by using image pro-

cessing without attached devices (Asai et al., 2002).

They evaluated the utility of the interface by com-

paring the interface with a joystick. The results in-

dicate that the performance of the interface is com-

parable to that of the joystick in terms of viewpoint

control, but enhances the sensation of speed. Man-

ders et al. presented a method for interacting with 3D

objects in an immersed 3D virtual environment with a

head-mounted display (HMD) by tracking user’s hand

gestures with a stereo camera (Manders et al., 2008).

The system allows the user to manipulate a 3D object

with ﬁve degrees of freedom using a series of intuitive

hand gestures. Steinicke et al. developed a VR-based

user interface for presence-enhancing gameplay with

which players can explore the game environment in

the most natural way, i. e., by real walking (Steinicke

et al., 2009). While the player walked through the

virtual game environment by wearing a HMD, they

analyzed the usage of transitional environments via

which the player could enter a virtual world. The re-

sults of the psychophysical experiments have shown

that players can be guided on circular arc with a ra-

dius of 22.03m whereas they believe themselves to be

walking straight. Oshita proposed a motion-capture-

based control framework for third-person view virtual

environments with a large screen (Oshita, 2006). The

framework can generate seamless transitions between

user controlled motion and system generated reactive

motions.

Because these systems use user’s body motions to

control view in virtual environment, there are a few

fundamental problems. A problem is that the view

control is rough by using body motions. Another

problem is that the device input makes the user feel

the seam between the real and virtual worlds. There-

fore, some systems to adopt eye movement for view

control have been developed.

Miyashita et al. proposed an electrooculography

(EOG)-based gaze interface that was implemented by

mounting EOG sensors on a HMD with a head-tracker

and proposed a gaze estimation method on the HMD

screen(Miyashita et al., 2008). The accuracy of gaze

estimation was experimentally determined to be 68.9

%. The system solves the problems of eye-tracker

devices’ block to a HMD. Haffegee et al. focusses

on methods and algorithms for using an eye-tracker

to takes the eye-tracker output and converts it into a

virtual world gaze vector in an immersive VE by us-

ing an eye camera and a head tracker (Haffegee et al.,

2007).

These systems mainly adopt the user’s line of

sight. However, people usually use their eye move-

ments to not only control line of sight but also several

controls for sight such as focus, gaze, interest and af-

fect. For instance, people usually tighten their eyelids

to look carefully. People also widen their eyes to look

at someone with interest or in surprise. Additionally,

because a person with x-ray vision in a science ﬁc-

tion movie tightens his/her eyelids to look through ob-

jects, we proposed that it is possible to enhance user’s

visual functions in a virtual environment by tracking

eye movements in detail.

3 A VIEW CONTROL METHOD

USING EMG OF EYELIDS

We proposed a view control method associated to

movements of use’s eyelids to enhance user’s vi-

sual functions in a virtual environment (TheAuthors,

2011). The basic method deals with three types

of eyelid movements: staring, neutral and widely-

opening. The neutral is a state of looking at some-

thing naturally without straining or widely-opening

(see ﬁgure 1a). The staring is a state of looking care-

fully with gathering eyebrows (see ﬁgure 1b). The

widely-opening is a state of looking at whole with

raising eyebrows (see ﬁgure 1c). Figure 2 shows the

state transition diagram of the three states.

By using the states, we designed two types

of view control techniques in a virtual environ-

ment: the zoom in/out control technique and the see

through/annotation control technique. By using the

APhysiologicalEvaluationofImmersiveExperienceofaViewControlMethodusingEyelidEMG

225

Figure 1: Three states of eyelid movements: (a) neutral, (b)

staring, and (c) widely-opening.

Figure 2: State-transition diagram of the three eyelid states.

zoom in/out technique, a user can look at a distant

object as the zoomed-in view by looking at it with

gathering eyebrows. Moreover, the user can look at

objects, which are outside of the user’s neutral view,

as the zoomed-out view by looking at the center of the

view with raising eyebrows. The view of zoom in/out

becomes the normal view when the user repositions

the eyebrows and relaxes the palpebral muscles.

On the other hand, by using see-

through/annotation technique, a user can look

inside an object such as an engine of a car by looking

at it with gathering eyebrows. Moreover, the user can

read annotation text, such as spec of a car around it

by looking at it with raising eyebrows. The views

become the initial view when the user repositions the

eyebrows and relaxes the palpebral muscles.

We think that the system improves the user’s im-

mersive experience because of the relationship be-

tween the actual eye movements and the visual feed-

back. This advantage solves the problem of reducing

a user’s immersive impression because of the tenuous

relationship between visual feedback and the move-

ment of conventional input devices, such as a mouse,

a joystick, or a location sensor.

Additionally, the system has a possibility to pro-

vide a user with controlling the view in multiple steps

by adjusting the strength of his/her eyelid muscle. In

other words, the stronger the user gathers his/her eye-

brows, the larger the zoom is, and the higher the trans-

parency is.

3.1 Implementation

We use a Z800 3D Visor (eMagin Co.) as a

head-mounted display (HMD), a ProComp Inﬁniti

(Thought Technology Ltd.) as a biofeedback device,

two MyoScan sensors (Thought Technology Ltd.) as

pre-ampliﬁed surface electromyography (EMG) sen-

sors, and a PC to implement an eyelid-movement rec-

ognizer and a 3 D virtual environment . The speci-

ﬁcations of the HMD are as follows: a 105-inch vir-

tual screen at a distance of 3.6 m, 40-degree angle

of view, horizontal 360-degree angle and vertical 60-

degree angle head tracking, 800 600 pixels. The

biofeedback device and the EMG sensors are used to

record the EMG signals on a user’s facial surface and

to send the data to the PC via USB. The recording

frequency is 20 Hz, and the recorded signals are raw

voltage values. The PC is used to process the EMG

data and to render the scene of a virtual environment

with Microsoft Visual C++ and OpenGL. The three

dimensional view of the scene is controlled with the 6

degrees of freedom data of the head-tracking HMD.

Figures 3 shows the HMD and the electrodes of

the EMG sensors attached to a user’s face. An EMG

sensor consists of three electrodes: positive, negative,

and reference. We use an EMG sensor to detect eyelid

movements because it is difﬁcult to detect it by image

data processing with a camera, such as a study of Val-

star et.al(Valstar et al., 2006), because of the narrow

space between user’s face and the HMD (Figure 3c).

Figure 3: State of wearing the HMD and attaching the EMG

sensors in our proposed system.

3.2 Measuring EMG and Detecting

Eyelid Movements

EMG is a technique for detecting and amplifying tiny

electrical impulses that are generated by muscle ﬁbers

when they contract. An EMG sensor can record the

signals from all the muscle ﬁbers within the recording

area of the sensor contact. Some research studies use

EMG to develop human interfaces. Manabe et al. pro-

posed the use of an EMG of facial muscles to control

an electric-powered wheelchair(Manabe et al., 2009).

Agustin et al. presented a low-cost gaze-pointing and

EMG-clicking device, which a user employs with an

EMG headband on his/her forehead (Agustin et al.,

2009). Clark et al. used the EMG on a user’s arm

movement to control a robotic arm (Jr. et al., 2010).

Also, Costanza st al. presented a formal evaluation

of subtle motionless gestures based on EMG signals

from the upper arm for interaction in a mobile con-

text(Costanza et al., 2007) , and Gibert et al. de-

PhyCS2014-InternationalConferenceonPhysiologicalComputingSystems

226

veloped describe a light expression recognition sys-

tem based on 8 facial EMG sensors placed on speciﬁc

muscles able to discriminate 6 expressions to enhance

human computer interaction (Gibert et al., 2009). It is

practical to use EMG data in order to enable a com-

puter to intuitively interact with body movements.

Our system measures the EMG data between the

eyebrows of a user and between the upper and lower

right side of a user’s right eye to detect three types of

eyelid movements: staring, neutral, and wide open-

ing (Figure 1). This means that the system measures

the EMG signals of the frontalis muscle to recognize a

wide opening state, and the EMG signals of the corru-

gator muscle to recognize a staring state. This is based

on our preliminary experiment (TheAuthors, 2011).

3.3 Calibration and Recognition

Typically, the values of the electric potential of the

EMG signals are positive and negative. Therefore,

our system converts all values into absolute values for

simple statistical processing for calibration and recog-

nition.

Our system calibrates the thresholds of the EMG

data to recognize the three types of eyelid movements

for an individual user, before the user uses the view

control method. After a user puts the EMG sensors

on his/her face, the system records the EMG data ﬁve

times among 20 frames at the two places while the

user maintains each of the eyelid movements. Next,

the system calculates the average and the standard de-

viation (SD) of the recorded EMG data of each eyelid

movement state of the user. Based on the calculations,

the threshold of the neutral state is the sum of the aver-

age and the SD of the state. The threshold of the star-

ing state is the difference between the average and the

SD of the state, and the threshold of the wide opening

state is the difference between the average and the SD

of the state.

While a user uses the eyelid-view-control method,

the system continuously records the EMG data among

four frames at the two places, calculates two averages

and two SDs from the data and compares the averages

and the SDs with the thresholds of the calibration data

in order to recognize the eyelid movements.

4 PHYSIOLOGICAL

EVALUATION

We conducted a physiological evaluation experiment

to compare the visually immersive experience of

the eyelid-movement input method with that of a

mouse input method as a conventional common in-

put method. We measured skin conductance (SC) of

a subject’s hand and blood-volume pulse (BVP) on

a subject’s ﬁnger and analyzing them. The reason is

that physiological data continuously and objectively

reﬂect emotions of users (subjects) who are experi-

encing an immersive virtual environment.

4.1 Experimental Task

The experimental task is to read a word and say it in

the air in front of the subject and on the road under the

subject’s foot from the edge of the top of a tall build-

ing in a 3D virtual environment. The subjects perform

the task by using our system or a mouse while wear-

ing the HMD for a ﬁrst person’s view with head track-

ing. Figure 4 illustrates the task. A subject stands at

the edge of the top of a tall building in a virtual en-

vironment. First, the subject reads a word and says it

on the road after looking down at the road by bending

his/her neck down and zooming in on the word (for

example, Figure 4 shows gHelloh). We set the font

size of the word to small so that it is impossible for

the subjects to read it without zooming it in. Next, the

subject reads a word and says it in the air in front of

him/her by raising his/her head (for example, Figure 4

shows gvirtualh). After that, the subject reads another

word and says it on the road again by zooming it in.

The subject repeats the task for ﬁve words.

Figure 4: Schematic representation of the experimental

task.

We set the task of reading a word from a high alti-

tude because we could provide the subjects with a fear

of the open high place in the 3D virtual environment,

and hypothesize that the more immersive feelings a

subject had, the more he/she would experience a fear

of the place. For the purpose of the more immersive

experience, we asked the subjects to stand at the edge

of the board that was approximately 4 cm thick (Fig-

ure 5).

APhysiologicalEvaluationofImmersiveExperienceofaViewControlMethodusingEyelidEMG

227

Figure 5: State of standing on a board to provide the subject

with the feeling of standing on the edge of a building.

4.2 Procedure

We covered the HMD with a black thick cloth to block

natural light from entering between the HMD and the

subject’s face, because the light in the real world de-

creases the subject’s immersive feelings in a virtual

world.

All subjects performed the task once using each

method (within subjects); the eyelid-movement input

method and a mouse input method. For the mouse in-

put method, a subject held a mouse on his/her thigh

while standing on the board. The left click button was

used to zoom in and the right click button was used

to zoom out. To avoid an ordering effect between

the two methods, half of the subjects performed the

task of eyelid movements ﬁrst and the other half per-

formed the task of mouse clicking ﬁrst.

The subjects practiced how to use each of the

methods for approximately 1 min before they started

to perform the task by using the assigned method.

During practice, the subjects conﬁrmed the head-

tracking function of the HMD and how to zoom in

and out by standing on the ground in the virtual en-

vironment, not by standing on the top of a building.

The subjects performed the experimental task at the

rate of approximately 2 min per method. We recorded

the subject’s physiological data of the time in order

to validate the differences in the physiological data of

a heightened sense of fear of the high place between

the eyelid-movement input method and the mouse in-

put method. We assumed these differences to reﬂect

the differences in immersive feelings between the in-

put methods.

After completing the tasks, the subjects answered

a questionnaire consisting of ﬁve-point scale pair

comparisons about the differences they perceived be-

tween the input methods. Table 1 shows the question-

naire.

4.3 Measurement of Physiological Data

We analyzed how the subjects immersed in the vir-

tual environment when using each of the methods by

measuring the skin conductance (SC) of a subject’s

hand and the blood-volume pulse (BVP) on a sub-

Table 1: Questionnaire for user’s experience.

Questions and indexes

Q1DFor which did you feel more immersive impression?