CONTROLLING A VIRTUAL BODY BY THOUGHT IN A

HIGHLY-IMMERSIVE VIRTUAL ENVIRONMENT

A Case Study in Using a Brain-Computer Interface in a Virtual-Reality Cave-like

System

Doron Friedman

VECG Lab, Dept. of CS, University College London, UK

Robert Leeb

Laboratory of Brain-Computer Interfaces, Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria

Larisa Dikovsky, Miriam Reiner

Department of Ed. In Technology and Science, Technion Touch-lab, Technion, Israel Institute of Technology, Haifa, Israel

Gert Pfurtscheller

Laboratory of Brain-Computer Interfaces, Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria

Mel Slater

ICREA-Universitat Politecnica de Catalunya, Departament de LSI Barcelona, Spain

and Department of Computer Science, UCL

Keywords:

Brain-computer interface, virtual reality, Cave, avatar, advanced user interfaces.

Abstract:

A brain-computer interface (BCI) can arguably be considered the ultimate user interface, where humans op-

erate a computer using thought alone. We have integrated the Anon-BCI into a highly immersive Cave-like

system. In this paper we report a case study where three participants were able to control their avatar using

only their thought. We have analyzed the participants’ subjective experience using an in-depth qualitative

methodology. We also discuss some limitations of BCI in controlling a virtual environment, and interaction

design decisions that needed to be made.

1 INTRODUCTION

Brain-computer interface (BCI) has been studied ex-

tensively as a tool for paralyzed patients, which may

augment their communication with the external world

and allow them better control of their limbs. However,

once it has been developed for these critical applica-

tions, we expect it will have profound implications on

many other types of user interfaces and applications.

BCI could be one of the most signiﬁcant steps fol-

lowing “direct manipulation interfaces” (Schneider-

man, 1983) – where intention is mapped directly into

interaction, rather than being conveyed through mo-

tor movements. Furthermore, if used in an immer-

sive virtual environment (IVE) this could be a com-

pletely novel experience and, in the future, lead to

unprecedented levels in the sense of presence (for re-

cent reviews of the concept of presence see (Vives and

Slater, 2005) and (Riva et al., 2003)).

A key requirement for a successful experience in

an immersive virtual environment (IVE) is the rep-

resentation of the participant, or its avatar (Pandzic

et al., 1997; Slater et al., 1994; Slater et al., 1998).

This paper describes the ﬁrst ever study where par-

ticipants control their own avatar using only their

thoughts. Three subjects were able to use the anon-

BCI to control an avatar, and their subjective experi-

ence was assessed using questionnaires and a semi-

structured interview. Naturally, a third-person avatar,

such as used in this experiment, is only one possible

interface to an IVE.

Using a BCI to control an IVE by thought raises

several major human-computer interaction (HCI) is-

sues: whether classiﬁcation of thought patterns is

continuous (asynchronous BCI) or only takes place in

speciﬁc moments (synchronous BCI), the number of

Fr iedman D., Leeb R., Dikovsky, L., Reiner M., Pfurtscheller G. and Slater M. (2007).

CONTROLLING A VIRTUAL BODY BY THOUGHT IN A HIGHLY-IMMERSIVE VIRTUAL ENVIRONMENT - A Case Study in Using a Brain-Computer

Interface in a Virtual-Reality Cave-like System.

In Proceedings of the Second International Conference on Computer Graphics Theory and Applications - AS/IE, pages 83-90

DOI: 10.5220/0002073100830090

 SciTePress

input classes recognized, the importance of feedback,

and the nature of the mapping between thoughts and

resulting action in the IVE. In this paper we refer to

these issues, and present a case study that speciﬁcally

addresses the issues of feedback and mapping.

A critical initial hypothesis is that natural map-

ping between thought processes and IVE function-

ality would improve the experience. A one-to-one

mapping seemingly makes intuitive sense, but hav-

ing this mapping is constraining because we are lim-

ited in the scope of thought patterns that we can

detect based on contemporary brain recording tech-

niques. In addition, it precludes other more com-

plex or more fanciful body image mappings; what

if we want to experiment with lobster avatars? (See

Jaron Lanier’s “everyone can be a lobster” statement

in http://www.edge.org/q2006/q06 7.html#lanier). In

the case study reported here we have found out that

natural mapping was reported to feel more natural and

easy than when the mapping was reversed. However,

the results do not indicate that BCI accuracy was bet-

ter with natural mapping than with reversed mapping.

The main implication of our case study is that this

new type of interface, whereby IVE participants con-

trol their avatars by thought, is possible, and should

be further pursued. In addition, we reveal new in-

sights about the HCI issues that are involved in such

an interface, and provide a ﬁrst glance into what the

experience of using such an interface may be like.

2 BACKGROUND

The possibility that people may be able to control

computers by thought alone, based on real-time anal-

ysis of electroencephalogram (EEG) waves, was al-

ready conceived as early as the 1970s (Vidal, 1973).

Recently, with advances in processing power, sig-

nal analysis, and neuro-scientiﬁc understanding of the

brain, there is growing interest in BCI, and a few suc-

cess stories. Current BCI research is focussing on

developing a new communication alternative for pa-

tients with severe neuromuscular disorders, such as

amyotrophic lateral sclerosis, brainstem stroke, and

spinal cord injury (Wolpaw et al., 2002).

Previous research has established that a BCI may

be used to control events within an IVE. One thread of

research tried to evaluate BCI as an additional control

channel for future combat pilots (Nelson et al., 1997;

Middendorf et al., 2000). Bayliss et al. studied BCI

based on the P3 evoked potential (EP) in the context

of a virtual reality driving simulation and a simulation

involving operating a few devices in a virtual apart-

ment (Bayliss and Ballard, 2000; Bayliss, 2003).

This previous research into IVE and BCI was all

based on several types of visually evoked responses.

Our research is based on a different BCI paradigm

that exploits motor imagination. Such a motor imag-

ination would, for example, be thinking about mov-

ing a hand or a foot, but without actually moving

it (Pfurtscheller and Neuper, 2001).

In a previous experiment (Anonymous, ), we have

allowed subjects to navigate a virtual street using BCI

in a Cave-like (Cruz-Neira et al., 1992) system. Our

results in that previous experiment provided some ev-

idence that a highly immersive environment such as a

Cave may not only improve user motivation, but may

also facilitate BCI accuracy. This suggests that there

is a great potential in using IVEs with BCI. However,

our research has also made us aware of the many lim-

itations and design issues that come into play when

using BCI as an interface to control and IVE, which

we now consider.

The ﬁrst issue is the number of different events (or

classes) distinguished in real-time, through the analy-

sis of EEG. As we add more classes, accuracy quickly

drops, and the number of EEG channels (recorded

brain areas) needs to grow, which makes the sessions

more complex and time consuming. Another limi-

tation is that BCI is often synchronous, or trigger-

based, i.e., the classiﬁcation is not applied continu-

ously, but only in speciﬁc time windows following an

external cue, such as a short sound after which partic-

ipants are required to have the appropriate thoughts.

Asynchronous BCI is possible, but accuracy is com-

promised (Millan and Mourino, 2003).

Wolpaw et al. highlight the importance of feed-

back for BCI (Wolpaw et al., 2002). In order to be

effective, the feedback needs to be immediate. How-

ever, providing continuous and immediate feedback

causes a problem. If we look at the accuracy of clas-

siﬁcation over time, we see that there is typically a

delay of 1-3 seconds between the onset of the trigger

and the optimal classiﬁcation. The typical approach,

which we also adopt here, is to provide immediate,

continuous feedback, for the whole classiﬁcation du-

ration (approximately four seconds), even though we

know the classiﬁcation is rarely correct throughout

this whole duration. Figure 1 shows the data from a

typical training run: in this case the classiﬁcation rate

reaches an optimum 2-3 seconds after the trigger, and

then drops again.

A natural question is whether a more direct map-

ping between the type of thought required to initiate

an action and the feedback from that thought would

improve BCI performance, and how this mapping

would be experienced by the participants.

GRAPP 2007 - International Conference on Computer Graphics Theory and Applications

0 1 2 3 4 5 6 7

t[sec]

Classification Error [%]

both

foot (1)

right (2)

Figure 1: Classiﬁcation error over time, averaged over 40

triggers in one run. A cross is displayed from time 0, and an

arrow cue is given at second 3 for a duration of 1.25 second,

which indicated to the subject what they should “think”.

3 METHOD

We have opted to use a basic anon-BCI paradigm,

which is binary and synchronous. This entails limited

information throughput, but increases subjects’ BCI

success rate, which is thus appropriate for a ﬁrst-of-

its-kind research.

3.1 Subjects and Training

Eleven subjects went through “traditional” (2D,

monitor-based) BCI training, and the top three were

selected for the actual IVE study. This is a typical

BCI screening process. It is known that a small per-

centage of population can easily adapt to the BCI and

a larger majority can reach similar accuracy levels,

but only with long periods of training (Guger et al.,

2003), thus typically 2-5 subjects are used to prove

the feasibility of a system. Since we were also inter-

ested in comparing between two conditions, we had

each subject repeat each condition four times.

Each subject ﬁrst took part in a number of training

runs without feedback. In each run the subject had to

imagine a movement of both their legs or a movement

of their right hand in response to a visual cue stimu-

lus presented on a computer monitor, in the form of

an arrow pointing downwards or to the right, respec-

tively (Figure 2). In addition to the visual cue an au-

ditory cue was also given either as a single beep (hand

imagination) or as a double beep (legs imagination).

Each trial started with a ﬁxation cross (second 0) fol-

lowed at second 3 by the cue-stimulus presented for

1.25 seconds. There was a random duration interval

of 0.5-2 seconds between the trials.

Figure 2: “Traditional” BCI in front of a monitor: the arrow

on the red cross indicates to the subject whether they should

imagine moving their hand or their feet. Subjects need to

keep concentrating on this thought as long as the cross is

displayed; for 4.25 seconds.

Forty EEG trials, twenty for every class, were

recorded in one run. The EEG trials from runs with-

out feedback were used to set up a classiﬁer for dis-

criminating between the two different mental states.

In further runs, visual feedback in the form of a mov-

ing bar was given to inform the subject about the ac-

curacy of the classiﬁcation during each imagination

task (i.e., classiﬁcation of right-hand imagination was

represented by the bar moving to the right, classiﬁca-

tion of feet movement imagination made the bar move

downward; see Figure 3). The training and feedback

runs were repeated a number of times over a period

of a few weeks, until a satisfactory classiﬁcation for

the discrimination between the two motor imagina-

tion tasks was achieved. Each session, composed of a

number of runs, lasted approximately two hours.

Figure 3: “Traditional” BCI in front of a monitor: the white

bar provides immediate and continuous feedback for 4.25

seconds.

Two subjects had a high BCI success rate in their

ﬁrst few runs: one had over 85% accuracy and the

other over 90% accuracy. Finding a third subject was

CONTROLLING A VIRTUAL BODY BY THOUGHT IN A HIGHLY-IMMERSIVE VIRTUAL ENVIRONMENT - A

Case Study in Using a Brain-Computer Interface in a Virtual-Reality Cave-like System

more difﬁcult. After a few weeks of training, three

other subjects reached approximately 75% accuracy,

showing improvement over time, but the improve-

ment was slow.

Eventually, the study proceeded with three sub-

jects: two females and one male (aged 21,49, and 27,

respectively). All subjects were right handed, without

a history of neurological disease. Subjects gave for-

mal consent to participate in the study, and were paid

the equivalent of $12 per hour. Each session lasted 3-

4 hours, and each subject went through two sessions.

3.2 EEG Recording

Three EEG channels were recorded bipolarly (two

electrodes for each channel). Electrode were placed

2.5 cm anterior and 2.5 cm posterior to positions

C3, C4, and Cz, of the “10-20 International System”,

which is a standard for electrode placement based on

the location of the cerebral cortical regions. The EEG

was ampliﬁed between 0.5 and 30 Hz by an EEG am-

pliﬁer (g.tec Guger Technologies, Graz, Austria) and

processed in real-time. Sampling frequency was 250

Hz.

3.3 Feature Extraction and

Classiﬁcation

BCI systems apply an online classiﬁcation to the EEG

signal. Two frequency bands selected from each EEG

channel served as features for the algorithm. The log-

arithmic band power was calculated in the alpha (8-

12 Hz) and beta (16-24Hz) bands over one-second

epochs. These features were classiﬁed with Fisher’s

linear discriminant analysis (LDA) and transformed

into a binary control signal.

3.4 The Virtual Environment

The study was carried out in a four-sided ReaCTor

system that is similar to a Cave (Cruz-Neira et al.,

1992), with Intersense IS900 head-tracking. The ap-

plications were implemented on top of the DIVE soft-

ware (Frecon et al., 2001; Steed et al., 2001).

The environment included two furnished virtual

rooms. The avatar was projected (using stereo dis-

play) to appear standing approximately half a meter

in front of the subject, who was sitting on a chair. The

avatars were matched for gender with the subject (see

Figure 4).

(a)

(b)

Figure 4: (a) A female subject and her avatar in the virtual

room. The subject is connected to the BCI equipment, in-

side the Cave-like system. (b) A male subject with the male

avatar, in the same setting.

3.5 Experimental Conditions

The visual feedback was different in two conditions.

In the ﬁrst condition, which we call the normal condi-

tion, the mapping between the thought pattern and re-

sult in the IVE was intuitive: when the subjects imag-

ined moving their right arm the avatar would wave

its right arm, and when they imagined moving their

legs the avatar would start walking forward slowly.

In the second condition the mapping was reversed:

when the subjects imagined moving their right arm

the avatar would start walking, and when the subjects

imagined moving their legs the avatar would wave its

arm. The feedback was continuous for the same du-

ration as in the monitor-based BCI training (4.25 sec-

onds). In both conditions, the audio triggers were the

same as in the training phase: single beep indicated

that the subjects need to think about their arm, and a

double beep indicated they need to think about their

legs.

4 RESULTS

4.1 BCI Accuracy

Each subject carried out four runs of both conditions,

thus eight runs in total. Each run included 40 trigger

GRAPP 2007 - International Conference on Computer Graphics Theory and Applications

events, and each trigger was followed by 80 classiﬁca-

tion results, one every approximately 50 milliseconds.

Thus, the data include 8 runs per subject, and each run

includes 3200 trials. BCI accuracy is determined by

the percentage of successful trials.

In order to test the signiﬁcance of the results we

carried out the equivalent of two-way analysis of vari-

ance, using the number of successes out of the 12800

trials in each of the conditions. In this analysis the

response variable is therefore taken as a binomial dis-

tribution (rather than Gaussian) and it is a standard

application of logistic regression. The results show

that there were highly signiﬁcant differences between

the three subjects (at a signiﬁcance level that is too

small to be quoted). Subject M1 had the highest suc-

cess rate (94%), subject F1 had the next highest (86%)

and subject F2 the lowest (81%) – and these are in

keeping with what is typically found in BCI experi-

ments. The raw ﬁgures show that in the normal con-

dition the success rate was 86.7% and in the reverse

condition 87.7% and with n = 12800 per condition

this difference is signiﬁcant. However, this does not

take into account the differences between the subjects

– since the very large advantage of the reverse con-

dition for subject F1 (88% reverse compared to 84%

normal) distorts the overall result. For subject M1 the

reverse condition is signiﬁcantly higher than the nor-

mal condition (z = -11.3,P = 0) for subject F2 there is

no signiﬁcant difference between the reverse and nor-

mal condition (z = 1.02, P = 0.31) and for subject F1

the normal condition is signiﬁcantly higher than the

reverse condition (z = 3.88, P = 1.0e-4). These are

carried out using a normal test for the difference be-

tween proportions. Thus, overall, no particular con-

clusion can be drawn one way or another about the

effectiveness of the mapping in terms of BCI perfor-

mance. Figure 5 depicts the performance of the three

subjects in the two conditions.

4.2 Qualitative Results

The control of a virtual body using thought is a com-

pletely new type of experience, and we were inter-

ested in getting some insight into the subjective ex-

periences of the subjects. We thus used a combina-

tion of questionnaires and semi-structured interviews.

The goal of the subjective questionnaires and inter-

views is exploratory. We hope to partially reconstruct

the subjective experience in order to gain insight into

this novel experience; this is a type of ideographic

study (Kelly and Main, 1978).

The subjective impressions of people, unlike their

BCI accuracy, is dependent on contingent factors such

as social background, video game exposure, etc. Be-

low we describe what the three subjects reported, but

a study with a larger number of subjects is necessary

in order to extrapolate this evidence.

After their ﬁrst IVE session, each subject com-

pleted several questionnaires: the SUS presence ques-

tionnaire (Slater et al., 1994), the Trinity question-

naire for body plasticity (TABP) (Desmond et al.,

2002), and a questionnaire regarding body projec-

tion: When a person has the sensation that an ob-

ject (whether real or virtual) is experienced as part of

his/her own body, this is referred to as ’body projec-

tion’. The most famous example of this is the rubber

arm illusion (Botvinick and Cohen, 1998; Armel and

Ramachandran, 2003). In order to evaluate whether

this type of body projection was experienced by our

subjects, we have also administered a questionnaire

recently designed in our lab for that purpose.

Figure 5: BCI error levels of the three subjects in the two

experimental conditions.

The questionnaires are comprised of 7-point or 5-

point Likert-scale questions. First, all questions were

“normalized” so that all low and high rates indicate

the same trend, e.g., low presence would always cor-

respond to a low rating. Then we counted how many

extreme (very low or very high) answers each subject

provides (For 7-point questions 1 and 2 were con-

sidered low and 6 and 7 were considered high, and

for 5-point questions only 1 and 5 were considered

extreme). By subtracting the number of high scores

from the number of low scores, we can classify the

result of that questionnaire into three categories: low,

high, or neutral. Our three subjects showed consis-

tency in their answers – there was no case where there

were both high and low scores for the same question-

naire. Table 1 summarizes the results, which were

also used to complement the interviews in gaining an

insight into the subject’s experience.

After completing the questionnaires, the subjects

went through a semi-structured interview. The inter-

views were audio-taped and transcribed. Such inter-

view agendas are designed in advance to identify logi-

cally ordered themes, following the recommendations

CONTROLLING A VIRTUAL BODY BY THOUGHT IN A HIGHLY-IMMERSIVE VIRTUAL ENVIRONMENT - A

Case Study in Using a Brain-Computer Interface in a Virtual-Reality Cave-like System

Table 1: Summary of questionnaire results. Result in each

category can be either high (+), low (-), or average (0).

Subject Plasticity Body projection Presence

F1 + + 0

M1 - - -

F2 - 0 +

of (Smith, 1977). We asked open-ended questions,

and intervention was minimized to occasional neutral

questions to encourage the subjects to continue.

4.2.1 IVE-BCI versus “Traditional” BCI

All subjects found the IVE-based BCI to be more en-

joyable and interesting. This is obvious, but may be

important in its own right: BCI training is typically

very long and exhausting; IVE may prove useful in

signiﬁcantly improving the training experience and

increase motivation.

Subjects F1 and F2 thought the IVE-based BCI

was easier (although they did not actually improve

their BCI performance). Subject F1 compared the

monitor-based BCI (which she refers to as a ‘little

line’) with the IVE experience: “I felt it was easier to

make her do things. Because something was actually

happening. Because when you’re thinking about your

feet but it’s just a little line whereas if you’re think-

ing about your feet and she moves it’s, I don’t why, it

just seemed make to more sense.” Subject M1, who

reported very low presence, mentioned the IVE was

more enjoyable.

4.2.2 Relationship with Avatar

Note that the subjects were not told that the virtual

body is intended to be their avatar, and in principle

there is no reason why people should associate this

virtual body with themselves. However, two of the

three subjects (M1 and F2) referred to the virtual body

as a puppet controlled by them, which is a typical way

to regard an avatar. The third subject (F1) even occa-

sionally referred to it in ﬁrst person.

Subject F1 seemed to have the highest level of

projection of her body to the avatar. This was not

only evident from the questionnaire, but also, dur-

ing the sessions. At ﬁrst, the subject referred to the

avatar as “she”, but after a few runs she started refer-

ring to it as “I”. In the questionnaires this subject re-

ported a medium level of presence, and a high degree

of body plasticity. In the interview, this subject said:

“Although I was controlling her, I wasn’t moving my

hand. and I’d know if I was moving my hand.” How-

ever, later she added: “..Oh yeah. It’s because I, my

brain, did move the hand. Towards the end I did feel

it was representing me. I always felt like it was repre-

senting me but I didn’t feel it was a part of me... It’s

difﬁcult. When you think about moving your hands

you know whether you’re moving your hands or not.

If she was moving her hand mine wasn’t moving. So

she can’t really be a part of me. -Cause to feel the

hand moving you’d have to feel the air going past it.

But the more you were in the more comfortable you

would become with that becoming you. It would just

be a different type of you. Like a different version

of you, almost. But it will never be you... First like

another body. Most of the time.”

Subject F2 reported higher presence but lower lev-

els of body plasticity and body projection. “I couldn’t

think of her as myself. I was trying to get into her

skin, it was frustrating when I couldn’t.. when i was

successful I was becoming closer, I was becoming

her. Or she was becoming me. I’m still saying that for

me to experience my movement somehow she was a

distraction. Thinking of movement I could have done

better without her. We didn’t click...But the connec-

tion was more like a puppet master...Get rid of her.

Just let me move in the environment - that was amaz-

ing. She was the task I was supposed to do.”

Subject M1 reported low presence and low body

projection. In the interview, he said: “First I thought it

was another person standing in front of me. I thought

what the objective was. I was wondering what would

happen to this person. I didn’t feel as it being my

body but I felt I had some control of the person or

of the body standing in front of me... I would best

describe it like a puppet.”

4.2.3 Mapping of Thoughts to Avatar Motion

Note that although we consider the mapping between

thought patterns and resulting avatar action to be nat-

ural in the normal condition, the mapping is not nec-

essarily perfect. For example, subjects were not in-

structed to think of a particular leg motion, and thus

they imagined cycling or kicking, whereas the avatar,

although it moved its feet, would perform a differ-

ent action: walking. This could be confusing for the

subjects, and might even divert them from the spe-

ciﬁc thought they have been trained with. In the in-

terviews, all subjects replied that this mismatch was

not a problem, and that the feedback seemed appro-

priate. We do not know if this mismatch affects the

BCI performance.

Subjects F1 and F2 (who experienced medium and

high presence, respectively) mentioned that the fact

that they were sitting and the avatar was standing was

more problematic. F2: “I usually do not walk forward

while I am sitting down..” We, of course, anticipated

this problem. In pilot runs we tried to have the sub-

GRAPP 2007 - International Conference on Computer Graphics Theory and Applications

ject stand in the Cave; this proved uncomfortable and

generated too many motion artifacts in the EEG sig-

nals. We could have had the avatar sit down, but that

was not the point of the study; in our vision for a fu-

ture you project your body onto the avatar’s body, and

then the avatar can be free to operate in the VE, con-

trolled by your thought.

When asked about the difference between the two

conditions, all subjects mentioned that they had to

concentrate more in the reverse condition. This may

be an explanation as to why they sometimes per-

formed better in the reverse condition than in the nor-

mal condition. F1: “It was confusing, but I didn’t

ﬁnd it difﬁcult.” F2: “I’m not sure if seeing the feed-

back was so confusing – just adding all these layers.

It made it difﬁcult to concentrate.” M1 reported very

low presence, yet mentioned: “I was surprised that

when it was reversed I found it harder to concentrate.

It made me confused. At all times, not only in the

beginning. So there must have been something on an-

other level – I must have been inﬂuenced.”

5 CONCLUSION

We have devised a system that allows people to be

able to control a virtual body in an IVE, with accuracy

ranging from 72% to 96%. We consider this to be a

proof of the feasibility of this innovative interface. We

have used qualitative methods to get a sense of this

new type of experience: what did it feel like? what

was the nature of the relationship between the subjects

and their avatars?

There is growing interest in the BCI community

to use IVE, and some evidence that IVE may assist

in BCI training, or even improve BCI performance.

Our ﬁnding suggests that BCI in IVE is more enjoy-

able than traditional BCI, and subjects ﬁnd it more in-

tuitive and natural. However, subjects did not seem

to perform better when the mapping between their

thoughts and the feedback was natural, as compared

to when this mapping was reversed. This is despite the

fact that the subjects did report that the reverse con-

dition seemed more confusing and less intuitive. The

results we describe in this paper thus indicate that the

story is complex and justify further research.

In 1965 Sutherland described the Ultimate Dis-

play system – which was the forebear of today’s

highly immersive virtual reality systems (Sutherland,

1965). In this paper we are pointing the way towards

the Ultimate Human-Computer Interface, an interface

through thought - as has been described in novels by

authors such as William Gibson (Gibson, 1984) and

Neal Stephenson (Stephenson, 1991). The research

described in this paper has shown that it is possible

to control a virtual body by thought, and has explored

performance-related results and the subjective experi-

ence that this entails. In future work we plan to take

this paradigm further, by trying to achieve a higher

level of body projection from participants to their

avatars, and by allowing the participants to achieve

a larger variety of tasks through their avatars, which

they control by thought. We hope this could be a ﬁrst

step towards an ultimate future interface.

ACKNOWLEDGEMENTS

This work has been supported by the European Union

FET projects PRESENCIA, IST-2001-37927, and

PRESENCCIA, IST-2006-27731. We would like to

thank Marco Gillies and his support with the charac-

ter animation. We are also grateful to David Swapp

and other members of the VECG lab in UCL for their

support.

REFERENCES

Armel, K. C. and Ramachandran, V. S. (2003). Project-

ing sensations to external objects: Evidence from skin

conductance response. Proc R Soc Lond B Biol Sci,

270:1499–1506.

Bayliss, J. D. (2003). Use of the evoked potential P3 com-

ponent for control in a virtual apartment. IEEE Trans.

Rehabil. Eng., 11(2):113–116.

Bayliss, J. D. and Ballard, D. H. (2000). A virtual reality

testbed for Brain Computer Interface research. IEEE

Trans. Rehabil. Eng., 8(2):188–190.

Botvinick, M. and Cohen, J. (1998). Rubber hands ’feel’

touch that eyes see. Nature, 391(6669):756.

Cruz-Neira, C., Sandin, D. J., DeFanti, T. A., Kenyon, R. V.,

and Hart, J. C. (1992). The CAVE: Audio visual ex-

perience automatic virtual environment. Comm. ACM,

35(6):65–72.

Desmond, D., Horgan, O., and MacLachlan, M. (2002).

Trinity Assessment of Body Plasticity (TABP). Depart-

ment of Psychology, Trinity College, Dublin.

Frecon, E., Smith, G., Steed, A., Stenius, M., and Stahl,

O. (2001). An overview of the COVEN platform.

Presence: Teleoperators and Virtual Environments,

10(1):109–127.

Friedman, D., Leeb, R., Guger, C., Steed, A., Pfertscheller,

G., and Slater, M. (2006). Navigating virtual reality by

thought: What is it like? to appear in: Teleoperators

and virtual environments.

Gibson, W. (1984). Neuromancer. Voyager.

Guger, C., Edlinger, G., Harkam, W., Niedermayer, I.,

and Pfurtscheller, G. (2003). How many people are

CONTROLLING A VIRTUAL BODY BY THOUGHT IN A HIGHLY-IMMERSIVE VIRTUAL ENVIRONMENT - A

Case Study in Using a Brain-Computer Interface in a Virtual-Reality Cave-like System

able to operate an EEG-based brain-computer inter-

face (BCI)? IEEE Trans. Neural Syst. Rehabil. Eng.,

11:145–147.

Kelly, F. D. and Main, F. O. (1978). Ideographic research in

individual psychology: Problems and solutions. The

Journal of Individual Psychology, 34(2).

Middendorf, M., McMillan, G., Calhoun, G., and Jones,

K. S. (2000). Brain-Computer Interface based on the

steady-state visual-evoked response. IEEE Trans. Re-

habil. Eng., 8(2):211–214.

Millan, J. D. R. and Mourino, J. (2003). Asynchronous BCI

and local neural classiﬁers: An overview of the adap-

tive brain interface project. IEEE Trans. Neural Syst.

Rehabil. Eng., 11:159–161.

Nelson, W., Hettinger, L., Cunningham, J., and Roe, M.

(1997). Navigating through virtual ﬂight environ-

ments using brain-body-actuated control. In Proc.

IEEE Virtual Reality Annual Intl Symp., pages 30–37.

Pandzic, I. S., Thalmann, N. M., Capin, T. K., and Thal-

mann, D. (1997). Virtual life network: A body-

centered networked virtual environment. Presence-

Teleoperators and Virtual Environments, 6:676–686.

Pfurtscheller, G. and Neuper, C. (2001). Motor imagery

and direct brain computer communication. Proc. of

the IEEE, 89(7):1123–1134.

Riva, G., Davide, F., and IJsselsteijn, W., editors (2003).

Being There: Concepts, effects and measurement of

user presence in synthetic environments. Ios Press.

Schneiderman, B. (1983). Direct manipulation: A step

beyond programming languages. IEEE Computer,

16(8):57–69.

Slater, M., Steed, A., McCarthy, J., and Maringelli, F.

(1998). The inﬂuence of body movement on subjec-

tive presence in virtual environments. Human Factors,

40:469–477.

Slater, M., Usoh, M., and Steed, A. (1994). Depth of pres-

ence in virtual environments. Presence: Teleoperators

and Virtual Environments, 11(3):130–144.

Smith, J. (1977). Semi-structured interviewing and qualita-

tive analysis. In Smith, J., Harre, R., and Langenhove,

L., editors, Rethinking Methods in Psychology, pages

9–26. Sage Publications, London.

Steed, A., Mortensen, J., and Frecon, E. (2001). Spelunk-

ing: Experiences using the DIVE system on CAVE-

like platforms. In Immersive Projection Technologies

and Virtual Environments, volume 2, pages 153–164.

Springer-Verlag/Wien.

Stephenson, N. (1991). Snowcrash. ROC Publishing.

Sutherland, I. (1965). The ultimate display. In Proc. IFIPS

Congress 1965, volume 2, pages 506–508.

Vidal, J. J. (1973). Toward direct brain-computer commu-

nication. Annual Review of Biophysics and Bioengi-

neering, 2:157–180.

Vives, M. S. and Slater, M. (2005). From presence to con-

sciousness through virtual reality. Nature Reviews

Neuroscience, 6(4):332–339.

Wolpaw, J. R., Birbaumer, N., McFarland, D. J.,

Pfurtscheller, G., and Vaughan, T. M. (2002). Brain

Computer Interfaces for communication an d control.

Clin. Neurophysiol., 113(6):767–791.

GRAPP 2007 - International Conference on Computer Graphics Theory and Applications