NeuRow: An Immersive VR Environment for Motor-Imagery

Training with the Use of Brain-Computer Interfaces and Vibrotactile

Feedback

Athanasios Vourvopoulos

1,2

, André Ferreira

and Sergi Bermudez i Badia

1,2

Faculdade das Ciências Exatas e da Engenharia, Universidade da Madeira,

Campus Universitário da Penteada, 9020-105 Funchal, Portugal

Madeira Interactive Technologies Institute, Polo Científico e Tecnológico da Madeira,

Caminho da Penteada, 9020-105 Funchal, Portugal

Keywords: Brain-Computer Interfaces, Motor Imagery, Virtual Reality, Serious Games.

Abstract: Motor-Imagery offers a solid foundation for the development of Brain-Computer Interfaces (BCIs), capable

of direct brain-to-computer communication but also effective in alleviating neurological impairments. The

fusion of BCIs with Virtual Reality (VR) allowed the enhancement of the field of virtual rehabilitation by

including patients with low-level of motor control with limited access to treatment. BCI-VR technology has

pushed research towards finding new solutions for better and reliable BCI control. Based on our previous

work, we have developed NeuRow, a novel multiplatform prototype that makes use of multimodal feedback

in an immersive VR environment delivered through a state-of-the-art Head Mounted Display (HMD). In this

article we present the system design and development, including important features for creating a closed

neurofeedback loop in an implicit manner, and preliminary data on user performance and user acceptance of

the system.

1 INTRODUCTION

Motor Imagery (MI) is the mental rehearsal of

movement -without any muscle activation- and is a

mental ability strongly related to the body or

‘embodied’ cognition (Hanakawa, 2015). MI appears

to largely share the control mechanisms and neural

substrates of actual movement both in action

execution and action observation (Eaves et al., 2014),

providing a unique opportunity to study neural

control of movement in either healthy people or

patients (Mulder, 2007; Neuper et al., 2009). Since

MI leads to the activation of overlapping brain areas

with actual movement, and because sensory and

motor cortices can dynamically reorganize (Lledo et

al., 2006; Rossini et al., 2003), MI constitutes an

important component for motor learning and

recovery. Hence, MI has important benefits and is

currently utilized as a technique in

neurorehabilitation for people with neurological

impairments (Dickstein et al., 2013).

MI offers an important basis for the development

of brain-to-computer communication systems called

Brain–Computer Interfaces (BCIs). BCIs are capable

of establishing an alternative pathway between the

brain and a computer or prosthetic devices (Wolpaw

et al., 2002) that could assist (assistive BCI) or

rehabilitate physically (restorative BCI) disabled

people and stroke survivors (Dobkin, 2007).

More recently, Virtual Reality (VR) feedback has

also been used in MI BCI training, offering a more

compelling experience to the user through 3D virtual

environments (Lotte et al., 2013a). The fusion of BCI

and VR (BCI-VR) allows a wide range of experiences

where participants can control various aspects of their

environment -either in an explicit or implicit manner-

, by using mental imagery alone (Friedman, 2015).

This direct brain-to-VR communication can induce

illusions mostly relying on the sensorimotor

contingencies between perception and action (Slater,

2009).

The idea of utilising BCIs in virtual rehabilitation

(virtual reality and tele-medicine for

neurorehabilitation), was fostered in order to

complement current VR rehabilitation strategies

(Bermudez i Badia and Cameirao, 2012; Lange et al.,

Vourvopoulos, A., Ferreira, A. and Badia, S.

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback.

DOI: 10.5220/0005939400430053

In Proceedings of the 3rd International Conference on Physiological Computing Systems (PhyCS 2016), pages 43-53

ISBN: 978-989-758-197-7

2012) where patients with low level of motor control

–such as those suffering of flaccidity or increased

levels of spasticity (Trompetto et al., 2014)- could not

benefit due to low range of motion, pain, fatique, etc.

The main challenge in the use of BCIs, regardless

of the BCI cost, lies in the lack of reliability and good

performance at the system level that inexperienced

users have (Vourvopoulos and Bermúdez i Badia,

2016) due to BCI “illiteracy” of users (inability of the

user to produce vivid mental images of movement

resulting in poor BCI performance) (Allison and

Neuper, 2010; Vidaurre and Blankertz, 2009).

Although previous studies have shown mixed results,

the combination of haptic and visual feedback seems

to increase the performance (Gomez-Rodriguez et al.,

2011; Hinterberger et al., 2004). It has been shown

that replacing the standard visual BCI feedback with

vibrotactile feedback does not interfere with the EEG

signal acquisition (Leeb et al., 2013) and also does not

impact negatively the classification performance

(Cincotti et al., 2007; Leeb et al., 2013). On the other

hand, it has been shown to have a positive effect on

visual workload measured in a multiple object

tracking task (MOT) where the data revealed

significant differences between visual or tactile

feedback (Gwak et al., 2014). It has also been shown

that with the use of haptic feedback, the user cam pay

more attention to the task instead of to the feedback

(Cincotti et al., 2007), and in (Jeunet et al., 2015)

users achieved higher scores in the vibrotactile

feedback setting. Vibrotactile feedback has also been

used in a hybrid BCI system (Yao et al., 2014), where

MI with selective sensation (SS) were used in order

to increase performance. On this system, equal

vibration is applied to both wrists of the user and

he/she has to imagine that the vibration to one of the

sides is stronger than the other (SS). SS combined

with MI increased the overall performance of the

system. In (Jeunet et al., 2015), it is also reported that

the vibrotactile feedback applied on the user's hand

significantly increases MI performance. In

(Leonardis et al., 2012) the use of vibrotactile

feedback directly applied to certain tendons is used to

convey the illusion of movement to the user, and in

conjunction with a virtual representation of the arm,

significantly increased the accuracy of a BCI system.

Further, recent findings with the use of virtual arms

have shown that the combination of motor priming

(physical rehearsal of a movement) preceding BCI-

VR MI training can improve performance as well as

the capacity to modulate and enhance sensorimotor

brain activity rhythms, important in rehabilitation

research (Vourvopoulos et al., 2015).

In addition, there is an increased need for

alternative motivational mechanisms and feedback

approaches for BCI systems (Lotte, 2012; Lotte et al.,

2013b). Previous research in learning, states that a

poorly designed feedback can actually deteriorate

motivation and impede successful learning (Shute,

2008) while providing extensive feedback to the user

can lead to efficient and high quality learning (Hattie

and Timperley, 2007). Lotte et al. recommended a set

of guidelines for a good instructional design in BCI

training, in which (1) the user should only be

presented with the correct classified action for

enhancing the feeling of competence; (2) provide a

simplified and intuitive task; (3) meaningful and self-

explanatory task; (4) challenging but achievable, with

feedback on progress of achievement; and finally (5)

in an engaging 3D virtual environment (Lotte et al.,

2013b).

To date, and to the best of our knowledge, there is

not a holistic approach in BCI MI training that

combines the advantages of different feedback

modalities (immersive VR environment, vibrotactile

feedback), training aproaches (motor priming

preceding motor observation) and motivational

mechanisms (game-like tasks). Further, in order to be

able to harness the benefits of BCI in

neurorehabilitation, two questions need to be

addressed: (a) how can we increase user performance

in BCI MI training, and (b) how can we maximize the

activation of the brain areas responsible for actual

movement. Answering these questions will enable the

appearance of novel BCI paradigms that will allow us

to promote more efficiently reorganization of

sensorimotor cortices of motor impaired patients

(such as for instance stroke), which ultimately can

lead to higher levels of recovery.

In this paper we describe the development and

pilot assessment of NeuRow, a novel BCI-VR

environment for MI training. NeuRow makes use of

multimodal feedback (auditory, haptic and visual) in

a VR environment delivered through an immersive

Head Mounted Display (HMD), integrated in a BCI

MI training task (left | right hand motor imagery).

Finally, NeuRow is available for different

platforms and is accessible for free at

http://neurorehabilitation.m-iti.org/bci/.

2 METHODOLOGY

2.1 Experimental Setup

The experimental setup was composed by a desktop

computer (OS: Windows 8.1, CPU: Intel® Core™ i5-

2400 at 3.10 GHz, RAM: 4GB DDR3 1600MHZ,

PhyCS 2016 - 3rd International Conference on Physiological Computing Systems

Graphics: AMD Radeon HD 6700), running the

acquisition software, the BCI-VR task, HMD, EEG

system, and a vibrotactile module.

EEG Acquisition

The BCI system consisted of 8 active electrodes

equipped with a low-noise biosignal amplifier and a

16-bit A/D converter at 256 Hz (g.MOBIlab+

biosignal amplifier, g.tec, Graz, Austria). The spatial

distribution of the electrodes followed the 10-20

system configuration (Klem et al., 1999) with the

following electrodes over the somatosensory and

motor areas: Frontal-Central (FC5, FC6), Central

(C1, C2, C3, C4), and Central-Parietal (CP5, CP6)

(Figure 1 a). The signal amplifier was connected via

bluetooth to the desktop computer for the EEG signal

acquisition. EEG data acquisition and processing was

performed through the OpenVibe platform (Renard et

al., 2010). Finally, the data from OpenVibe was

transmitted to the RehabNet Control Panel

(Reh@Panel) (Vourvopoulos et al., 2013) via the

VRPN protocol (Taylor et al., 2001) to control the

virtual environment. The RehabNet Control Panel is

a free tool that acts as a middleware between multiple

interfaces and virtual environments.

Feedback Presentation

For delivering feedback to the user, the Oculus Rift

DK1 HMD was used (Oculus VR, Irvine, California,

USA). The HMD is made of one 7" 1280x800 60 Hz

LCD display (640x800 resolution per eye), one

aspheric acrylic lens per eye, 110° Field of View

(FOV), internal tracking through a gyroscope,

accelerometer, and magnetometer, with a tracking

frequency of 1000Hz (Figure 1 b).

Vibrotactile Feedback

A custom vibrotactile feedback module was

developed with out-of-the-box components including

an Arduino Mega 2560 board and vibrating motors.

The vibrating motors (10mm diameter, 2.7mm thick)

performed at 11000 RPM at 5V and were mounted on

cylindrical tubes that acted as grasping objects for

inducing the illusion of movement during the BCI

task (Figure 1 c). In our setup, a pair of carton-based

tubes with 12cm of length and 3cm diameter were

used. Finally, 3D printed cases were produced to

accommodate the vibrating motors inside the tubes.

All hardware and software blueprints are made

available for free online.

2.2 BCI Task Design

BCI-VR Training Protocol

The training protocol was designed and adapted based

on the Graz-BCI paradigm (Pfurtscheller et al., 2003),

substituting the standard feedback presented

(directional arrows) by multimodal VR feedback. The

first step of the training consist on the acquisition of

the raw EEG data in order to train a linear

discriminant classifier to distinguish Right and Left

imagined hand movements. Throughout the training

session, the user has to perform mental imagery of the

corresponding hand (based on the presented stimuli).

For each hand, the user is stimulated both visually

(VR action observation) and haptically through the

vibration on the corresponding hand (Figure 2 a). The

training session was configured to acquire data in 20

blocks (epochs) per class (Right or Left hand

imagery) in a randomized order. Following the

training, the data is used to compute a Common

Spatial Patterns (CSP) filter, a spatial filter that

maximizes the difference between the signals of the

two classes. Finally, the raw EEG and the spatial filter

are used to train a Linear Discriminant Analysis

(LDA) classifier.

Figure 1: Experimental setup (a) EEG cap with 8 active

electrodes, (b) HMD, (c) vibrotactile modules, (d) BCI

feedback.

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback

Figure 2: Neurofeedback loop. (a) During the training session, the user is performing in a randomized order MI combined

with motor observation of the virtual hands rowing while vibrotactile feedback is delivered to the corresponding hand. (b)

The user relies on MI alone in order to control the virtual hands in a closed-loop system after training.

BCI-VR Task

The BCI-VR task was designed based on literature

and previous work, incorporating important features

for a successful brain-to-computer interaction in

terms of feedback, protocol design, and accessibility.

The BCI-VR task involves boat rowing through

mental imagery only with the goal of collecting as

many flags as possible in a fixed amount of time.

NeuRow is a self-paced BCI neurogame, meaning

that is not event related, and the user controls the

timing of rowing actions like he/she would do in real-

life (Figure 2 b). NeuRow is a multiplatform virtual

environment developed in Unity game engine (Unity

Technologies, San Francisco, California, USA).

Finally, NeuRow is optimized for different platforms,

however with different features (Table 1). Namely:

- Desktop: The standalone version for PC,

supports high quality graphics for an immersive VR

experience with the support of the Oculus Rift DK1

headset, the Leap Motion hand controller (Motion

control, San Francisco, California, USA) available for

optional motor-priming before the MI BCI session.

Finally, vibrotactile feedback is supported through

the use of custom made hardware for controlling

through USB up to 6 vibration motors. Data logging

is supported for boat trajectory, target location, score

and time.

- Mobile: The mobile version is built for

Android OS devices, receiving data via the RehabNet

UDP protocol through the Reh@panel. For phones,

the VR feature is utilized for VR glasses (e.g. Google

Cardboard) by applying lens correction for each eye,

and using the phone gyroscope and magnetometer for

head tracking, offering an immersive experience

similar to the Oculus DK1 and DK2 HMDs.

- Web browser: The web version uses the Unity

web player (compatible through Internet Explorer,

Firefox or Opera), does not support the networking,

HMD and haptic components due to security

restrictions. Instead, the web NeuRow acquires data

through emulated keyboard events generated by the

Reh@panel.

The in-game interface is simple, with two high

fidelity virtual arms to rotate the oars, time indication,

score and navigational aids (Figure 3). NeuRow can

be customized with different settings, depending on

the experimental setup, BCI paradigm and running

platform. Through the settings, one can chose if the

session is part of MI training or self-paced online

control session for navigation of the boat. During

Table 1: NeuRow features for the different supported

platforms.

Features/

Platform

Desktop Android Web

Logging ✓ X X

VR ✓

(Oculus)

✓

(Google

Cardboard)

Hand

Tracking

✓

(Leap

Motion)

X X

Networking ✓ ✓ X

Platform

Independent

X X ✓

Vibrotactile

Feedback

✓

(Arduino)

X X

PhyCS 2016 - 3rd International Conference on Physiological Computing Systems

Figure 3: In-game interface. An arrow indicates the direction of the target and also the distance by changing its colour (red

for far blending up to green for close). Top Left: Remaining time for the end of the session. Middle: A flag with a ray acts as

the game targets, Top Right: Game scoring, counting the amount of targets.

training, the navigational arrow and the targets are

removed to focus the user only on the multimodal MI

BCI-VR task. During self-paced mode, the behaviour

of the boat can be changed by setting the heading

speed, turn speed and cut-off angle. The cut-off angle

is the allowed angle that the boat can turn with respect

to the target flag before stopping. This serves as a

protection mechanism to help the player not to

deviate in excess from the target.

2.3 Participants

A voluntary sample of 13 users (mean age of 28 ± 5

years old) was recruited for the pilot study, based on

their motivation to participate in the study. All

participants were male and right handed with no

previous known neurological disorder, nor previous

experience in BCIs. Participants were either

university students or academic staff. Finally, all

participants provided their written informed consent

before participating in the study.

2.4 Questionnaires

Before each BCI training session, demographics and

user data were gathered through the following

questionnaires:

- The Vividness of Movement Imagery

Questionnaire-2 (VMIQ2) was used to assess the

capability of the participant to perform an

imagined movement (Kinesthetic Imagery)

(Roberts et al., 2008). Kinesthetic Imagery (KI)

questions were combined with mental

chronometry by measuring the response time in

perceptual-motor tasks with the help of a timer.

- For assessing gaming experience we used the

Gamer Dedication (GD) questionnaire, a 15 factor

classification questionnaire in which participants

are asked whether they "strongly disagree," or

"strongly agree" with a series of statements about

their gaming habits (Adams and Ip, n.d.).

After the BCI task, the following questionnaires

were administered:

- The NASA TLX questionnaire was used to

measure task load considering Mental Demand,

Physical Demand, Temporal Demand,

Performance, Effort and Frustration (Hart and

Staveland, 1988).

- The core modules of the Game Experience

Questionnaire (GEQ) were used at the end of the

BCI session. GEQ assesses game experience

using Immersion, Flow, Competence, Positive

and Negative Affect, Tension, and Challenge

(IJsselsteijn et al., 2008).

- The System Usability Scale (SUS) is a ten-item

scale giving a global view of subjective

assessments of usability (Brooke, 1996).

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback

Figure 4: Ranked accuracy of performance in pure MI based BCI studies using two-classes (left and right hand imagery) with

respect to LDA classification (Boostani and Moradi, 2004; Garcia et al., 2003; Obermaier et al., 2001; Solhjoo and Moradi,

2004). The asterisk (*) over 4,5,7,8,9,10 and 12 (Vourvopoulos et al., 2015; Vourvopoulos and Bermúdez i Badia, 2016)

indicates studies which use the same feature extraction method (BP with CSP). The data of this study corresponds to the 4

best.

2.5 Data Analysis

Power Spectral Density (PSD)

EEG signals were processed in Matlab (MathWorks

Inc., Massachusetts, US) with the EEGLAB toolbox

(Delorme and Makeig, 2004) for extracting the Power

Spectral Density (PSD). The power spectrum was

extracted for the following frequency rhythms: Alpha

(8 Hz - 12 Hz), Beta (12 Hz - 30 Hz), Theta (4 Hz - 7

Hz), and Gamma (25 Hz - 90 Hz). Independent

Component Analysis (ICA) was used for removing

major artefacts related with power-line noise, eye

blinking, ECG and EMG activity. For the current

analysis, and because we were only measuring from

sensory-motor areas, data were averaged for all the

channels for each experimental condition.

Engagement Index

The Engagement Index (EI) is a metric proposed at

NASA Langley for evaluating operator engagement in

automated tasks, was validated through a bio-cybernetic

system for Adaptive Automation (Pope et al., 1995), and

is widely used in EEG studies for assessing engagement

(Berka et al., 2007). We therefore computed

engagement index from the EEG power spectrum

according to equation: EI = β/(α+θ), where α = Alpha

band, β = Beta band and θ = Theta band.

3 RESULTS

In the following section we analyse NeuRow’s BCI

task performance in terms of classifier score during

training, user acceptance as assessed by the SUS,

GEX and TLX questionnaires, and finally the

relationship between game behaviour and user

experience through the questionnaires and also the

EEG activity.

3.1 Performance

Comparing the performance score with previous

studies which used LDA classifiers in two class (left,

right hand) MI, we are able to gain insights

concerning the effectiveness of our BCI-VR

paradigm in terms of user control (Boostani and

Moradi, 2004; Garcia et al., 2003; Obermaier et al.,

2001; Solhjoo and Moradi, 2004). As illustrated in

Figure 4, the comparison places NeuRow as the

fourth highest with a mean performance of 70.7% out

of 12 studies. Moreover, of those studies that used

exactly the same feature extraction technique of band

power (BP) and CSP (Vourvopoulos et al., 2015;

Vourvopoulos and Bermúdez i Badia, 2016),

NeuRow scores the highest. Finally, of those studies

that used VR as a training environment

(Vourvopoulos et al., 2015), again NeuRow scores

first.

3.2 User Acceptance

To assess different aspects of the user experience

during online control of NeuRow, the mental

workload, gaming experience and system usability

were assessed after the task.

PhyCS 2016 - 3rd International Conference on Physiological Computing Systems

For workload, the NASA-TLX mean score was

relatively high at 66.8/100 (SD = 14.5). As it is

illustrated in Figure 5, the two lowest scores are those

for physical (M = 4.4, SD = 3.4) and temporal (M =

6.5, SD = 3) demand. The highest score is on effort

(M = 16.4, SD = 5.2) followed closely by frustration

(M = 13.3, SD = 5.2) and mental demand (M = 12.8,

SD = 5). Performance lies in the middle (M = 11.4,

SD = 6.2).

From the GEQ, we extracted seven domains based

on the sub-scale scoring. The highest score is in flow

(M = 3.1, SD = 0.4) followed by immersion (M = 2.8,

SD = 0.4) and positive affect (M = 2.8, SD = 0.7). A

moderate score is achieved on tension/annoyance (M

= 2.5, SD = 0.9) and challenge (M = 2.5, SD = 0.5).

Finally, competence (M = 1.8, SD = 0.7) and negative

affect scored the lowest (Figure 6).

Figure 5: TLX scores between 1-20 for mental demand,

physical demand, temporal demand, performance, effort

and frustration.

Figure 6: Scores for the GEQ core questionnaire domains.

The system usability assessed by the SUS scored

a mean of 74 (SD = 7.2). Based on the SUS rating

scale (Figure 7), our system is classified as “Good”

and it is within the acceptability range (Bangor et al.,

2009).

3.3 User-Profile and in-Game

Behaviour

By assessing the relationship of the reported

experience and the EEG activity with the in-game

behaviour (score, distance, speed, trajectory) we

Figure 7: SUS results for all users. Acceptability scales are

displayed on top (not acceptable, marginal and acceptable),

followed by the grade scale (A to F) and the adjective rating

(0-100).

Table 2: Correlation table between reported experience,

extracted EEG bands and in-game behaviour.

Distance Speed Score Smooth

ness

TLX:

Total

-.695 -.699 -.697

TLX:

Perfor

mance

-.595 -.599

TLX:

Frustr

ation

-.728 -.737 -.686

Mental

Chron

ometry

.618 .615 .728

Alpha

band

-.611 -.607

Theta

band

-.672 -.670

Engage

ment

Index

-.770 -.768 -.649 -.595

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback

identified a set of correlations. As illustrated in Table

2, the total workload correlates with distance, speed

and score. In addition, two TLX sub-domains have

correlations. Performance is significantly correlated

with distance and speed, as well as frustration is

significantly correlated with distance, speed and

score. Furthermore, mental chronometry (the

response time in perceptual-motor tasks),

significantly correlates with distance, speed and

score. Finally, from the extracted EEG bands and the

resulting Engagement Index, we can see that Alpha

and Theta bands are reversely correlated with

distance and speed. Finally, Engagement Index is

interestingly correlated with all in-game metrics. In

particular for distance, speed, score and trajectory

smoothness.

4 CONCLUSIONS

In this paper we presented the design, development

and pilot evaluation of NeuRow, a novel BCI-VR

system for MI training. In terms of classification

performance, the NeuRow BCI training paradigm

showed a high performance, scoring the first amongst

other studies with similar feature extraction and

classification methodologies. These data supports a

positive effect of the combination of immersive VR

and vibrotactile feedback to help users to produce

vivid MI (resulting in more distinct activation of

sensorimotor areas of the brain), which in turn that

can lead to increased performance and learning

(Sigrist et al., 2013). Furthermore, from the user

experience point of view, we can see high mental

effort as given by the TLX scales and low physical

and temporal demands. Previous research in

distinguishing difficulty levels with brain activity

measurements indicated an average mental workload

index of 26 (SD = 12.9) for the easy level, and 69 (SD

= 7.9) for the hard level (Girouard et al., 2009). The

combination of low physical demand (useful in low

mobility patients), increased effort (a conscious

exertion of power) and good classification

performance (better control that can lean in goal

achievement), constitutes a very promising finding

for the incorporation of this technology in stroke

rehabilitation, providing new possibilities for

rehabilitation programs. Moreover, increased flow

and immersion to the task, in combination with

increased positive affect, are good elements for

enjoyment of NeuRow that can be capitalized on to

further motivate and engage users in their BCI

training. From the correlation analysis between user

experience -subjectively measured through

questionnaires but also objectively measured through

EEG activity- and in-game behaviour, we can see that

people with increased workload will perform worse.

Interestingly, we can see that users with fast response

time in MI ability (as extracted from the mental

chronometry assessment) performed better in the

game, being it then an indicator of increased

capability of MI. Having a fast and vivid sensation of

kinesthetic imagery can be related to an increased

modulation of sensorimotor rhythms (Neuper et al.,

2005), resulting in better BCI calibration and hence

higher in-game performance. In addition, the reverse

correlation of the Engagement index with all the in-

game variables shows an important connection

between user engagement and in-game behaviour.

This relationship can help in developing a

neurofeedback closed loop were the engagement of

the user is used to adjust parameters of the game. This

would allow a dynamic adjustment of the game based

on user performance and cognitive state. This could

provide (1) a major assistance for new users and/or

neurologically impaired people and (2) reduced levels

of frustration and workload.

Overall, we showed that NeuRow, combining the

use of immersive VR environment, sensory

stimulation and motor-priming features, can provide

a holistic approach towards MI driven BCIs. In this

study we showcased user performance, user

acceptance and important features for closing the loop

in an implicit manner. Finally, NeuRow’s features

show promise and potential to be used for MI training

in stroke motor rehabilitation. Future work will

include a study with stroke patients with the ultimate

goal to clinically validate NeuRow in a longitudinal

MI-BCI study with functional brain imaging.

ACKNOWLEDGEMENTS

This work was supported by the European

Commission through the RehabNet project -

Neuroscience Based Interactive Systems for Motor

Rehabilitation - EC (303891 RehabNet FP7-

PEOPLE-2011-CIG), by the Fundação para a Ciência

e Tecnologia (Portuguese Foundation for Science and

Technology) through SFRH/BD/97117/2013, and

LARSyS (Laboratório de Robótica e Sistemas em

Engenharia e Ciência) through

UID/EEA/50009/2013. Software and instructions are

available online in the following address:

http://neurorehabilitation.m-iti.org/bci/.

PhyCS 2016 - 3rd International Conference on Physiological Computing Systems

REFERENCES

Adams, E., Ip, B., n.d. From Casual to Core: A Statistical

Mechanism for Studying Gamer Dedication [WWW

Document]. URL http://www.gamasutra.com/view/fea

ture/131397/from_casual_to_core_a_statistical_.php

(accessed 1.5.15).

Allison, B.Z., Neuper, C., 2010. Could Anyone Use a BCI?,

in: Tan, D.S., Nijholt, A. (Eds.), Brain-Computer

Interfaces, Human-Computer Interaction Series.

Springer London, pp. 35–54.

Bangor, A., Kortum, P., Miller, J., 2009. Determining What

Individual SUS Scores Mean: Adding an Adjective

Rating Scale. J Usability Stud. 4, 114–123.

Berka, C., Levendowski, D.J., Lumicao, M.N., Yau, A.,

Davis, G., Zivkovic, V.T., Olmstead, R.E., Tremoulet,

P.D., Craven, P.L., 2007. EEG Correlates of Task

Engagement and Mental Workload in Vigilance,

Learning, and Memory Tasks. Aviat. Space Environ.

Med. 78, B231–B244.

Bermudez i Badia, S., Cameirao, M.S., 2012. The

Neurorehabilitation Training Toolkit (NTT): A Novel

Worldwide Accessible Motor Training Approach for

At-Home Rehabilitation after Stroke. Stroke Res. Treat.

2012. doi:10.1155/2012/802157.

Boostani, R., Moradi, M.H., 2004. A new approach in the

BCI research based on fractal dimension as feature and

Adaboost as classifier. J. Neural Eng. 1, 212–217.

doi:10.1088/1741-2560/1/4/004.

Brooke, J., 1996. SUS-A quick and dirty usability scale.

Usability Eval. Ind. 189, 194.

Cincotti, F., Kauhanen, L., Aloise, F., Palomäki, T.,

Caporusso, N., Jylänki, P., Mattia, D., Babiloni, F.,

Vanacker, G., Nuttin, M., Marciani, M.G., Millán, J. del

R., 2007. Vibrotactile Feedback for Brain-Computer

Interface Operation. Comput. Intell. Neurosci. 2007,

e48937. doi:10.1155/2007/48937.

Delorme, A., Makeig, S., 2004. EEGLAB: an open source

toolbox for analysis of single-trial EEG dynamics

including independent component analysis. J.

Neurosci. Methods 134, 9–21. doi:10.1016/j.jneumeth

.2003.10.009.

Dickstein, R., Deutsch, J.E., Yoeli, Y., Kafri, M., Falash,

F., Dunsky, A., Eshet, A., Alexander, N., 2013. Effects

of Integrated Motor Imagery Practice on Gait of

Individuals With Chronic Stroke: A Half-Crossover

Randomized Study. Arch. Phys. Med. Rehabil. 94,

2119–2125. doi:10.1016/j.apmr.2013.06.031.

Dobkin, B.H., 2007. Brain-computer interface technology

as a tool to augment plasticity and outcomes for

neurological rehabilitation. J. Physiol. 579, 637–642.

doi:10.1113/jphysiol.2006.123067.

Eaves, D.L., Haythornthwaite, L., Vogt, S., 2014. Motor

imagery during action observation modulates automatic

imitation effects in rhythmical actions. Front. Hum.

Neurosci. 8. doi:10.3389/fnhum.2014.00028.

Friedman, D., 2015. Brain-Computer Interfacing and

Virtual Reality, in: Nakatsu, R., Rauterberg, M.,

Ciancarini, P. (Eds.), Handbook of Digital Games and

Entertainment Technologies. Springer Singapore, pp.

1–22.

Garcia, G.N., Ebrahimi, T., Vesin, J., 2003. Support vector

EEG classification in the Fourier and time-frequency

correlation domains, in: First International IEEE EMBS

Conference on Neural Engineering, 2003. Conference

Proceedings. Presented at the First International IEEE

EMBS Conference on Neural Engineering, 2003.

Conference Proceedings, pp. 591–594. doi:10.1109/C

NE.2003.1196897.

Girouard, A., Solovey, E.T., Hirshfield, L.M., Chauncey,

K., Sassaroli, A., Fantini, S., Jacob, R.J.K., 2009.

Distinguishing Difficulty Levels with Non-invasive

Brain Activity Measurements, in: Gross, T., Gulliksen,

J., Kotzé, P., Oestreicher, L., Palanque, P., Prates, R.O.,

Winckler, M. (Eds.), Human-Computer Interaction –

INTERACT 2009, Lecture Notes in Computer Science.

Springer Berlin Heidelberg, pp. 440–452.

Gomez-Rodriguez, M., Peters, J., Hill, J., Schölkopf, B.,

Gharabaghi, A., Grosse-Wentrup, M., 2011. Closing

the sensorimotor loop: haptic feedback facilitates

decoding of motor imagery. J. Neural Eng. 8, 036005.

doi:10.1088/1741-2560/8/3/036005.

Gwak, K., Leeb, R., Millan, J.D.R., Kim, D.-S., 2014.

Quantification and reduction of visual load during BCI

operation, in: 2014 IEEE International Conference on

Systems, Man and Cybernetics (SMC). Presented at the

2014 IEEE International Conference on Systems, Man

and Cybernetics (SMC), pp. 2795–2800. doi:10.1109/

SMC.2014.6974352.

Hanakawa, T., 2015. Organizing motor imageries.

Neurosci. Res. doi:10.1016/j.neures.2015.11.003.

Hart, S.G., Staveland, L.E., 1988. Development of NASA-

TLX (Task Load Index): Results of Empirical and

Theoretical Research, in: Meshkati, P.A.H. and N.

(Ed.), Advances in Psychology, Human Mental

Workload. North-Holland, pp. 139–183.

Hattie, J., Timperley, H., 2007. The Power of Feedback.

Rev. Educ. Res. 77, 81–112. doi:10.3102/003465430

298487.

Hinterberger, T., Neumann, N., Pham, M., Kübler, A.,

Grether, A., Hofmayer, N., Wilhelm, B., Flor, H.,

Birbaumer, N., 2004. A multimodal brain-based

feedback and communication system. Exp. Brain Res.

154, 521–526. doi:10.1007/s00221-003-1690-3.

IJsselsteijn, W., Poels, K., de Kort, Y.A., 2008. The Game

Experience Questionnaire: Development of a self-

report measure to assess player experiences of digital

games. TU Eindh. Eindh. Neth.

Jeunet, C., Vi, C., Spelmezan, D., N’Kaoua, B., Lotte, F.,

Subramanian, S., 2015. Continuous Tactile Feedback

for Motor-Imagery Based Brain-Computer Interaction

in a Multitasking Context, in: Abascal, J., Barbosa, S.,

Fetter, M., Gross, T., Palanque, P., Winckler, M. (Eds.),

Human-Computer Interaction – INTERACT 2015,

Lecture Notes in Computer Science. Springer

International Publishing, pp. 488–505.

Klem, G.H., Lüders, H.O., Jasper, H.H., Elger, C., 1999.

The ten-twenty electrode system of the International

Federation. The International Federation of Clinical

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback

Neurophysiology. Electroencephalogr. Clin.

Neurophysiol. Suppl. 52, 3–6.

Lange, B., Koenig, S., Chang, C.-Y., McConnell, E., Suma,

E., Bolas, M., Rizzo, A., 2012. Designing informed

game-based rehabilitation tasks leveraging advances in

virtual reality. Disabil. Rehabil. 34, 1863–1870.

Leeb, R., Gwak, K., Kim, D.-S., del R Millán, J., 2013.

Freeing the visual channel by exploiting vibrotactile

BCI feedback. Conf. Proc. Annu. Int. Conf. IEEE Eng.

Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf.

2013, 3093–3096. doi:10.1109/EMBC.2013.6610195.

Leonardis, D., Frisoli, A., Solazzi, M., Bergamasco, M.,

2012. Illusory perception of arm movement induced by

visuo-proprioceptive sensory stimulation and

controlled by motor imagery, in: 2012 IEEE Haptics

Symposium (HAPTICS). Presented at the 2012 IEEE

Haptics Symposium (HAPTICS), pp. 421–424.

doi:10.1109/HAPTIC.2012.6183825.

Lledo, P.-M., Alonso, M., Grubb, M.S., 2006. Adult

neurogenesis and functional plasticity in neuronal

circuits. Nat. Rev. Neurosci. 7, 179–193. doi:10.1038

/nrn1867.

Lotte, F., 2012. On the need for alternative feedback

training approaches for BCI. Presented at the Berlin

Brain-Computer Interface Workshop.

Lotte, F., Faller, J., Guger, C., Renard, Y., Pfurtscheller, G.,

Lécuyer, A., Leeb, R., 2013a. Combining BCI with

Virtual Reality: Towards New Applications and

Improved BCI.

Lotte, F., Larrue, F., Mühl, C., 2013b. Flaws in current

human training protocols for spontaneous Brain-

Computer Interfaces: lessons learned from instructional

design. Front. Hum. Neurosci. 7, 568. doi:10.3389/fn

hum.2013.00568.

Mulder, T., 2007. Motor imagery and action observation:

cognitive tools for rehabilitation. J. Neural Transm.

114, 1265–1278. doi:10.1007/s00702-007-0763-z.

Neuper, C., Scherer, R., Reiner, M., Pfurtscheller, G., 2005.

Imagery of motor actions: differential effects of

kinesthetic and visual-motor mode of imagery in single-

trial EEG. Brain Res. Cogn. Brain Res. 25, 668–677.

doi:10.1016/j.cogbrainres.2005.08.014.

Neuper, C., Scherer, R., Wriessnegger, S., Pfurtscheller, G.,

2009. Motor imagery and action observation:

Modulation of sensorimotor brain rhythms during

mental control of a brain–computer interface. Clin.

Neurophysiol. 120, 239–247. doi:10.1016/j.clinph.20

08.11.015.

Obermaier, B., Guger, C., Neuper, C., Pfurtscheller, G.,

2001. Hidden Markov models for online classification

of single trial EEG data. Pattern Recognit. Lett.,

Selected Papers from the 11th Portuguese Conference

on Pattern Recognition - RECPAD2000 22, 1299–

1309. doi:10.1016/S0167-8655(01)00075-7.

Pfurtscheller, G., Neuper, C., Müller, G.R., Obermaier, B.,

Krausz, G., Schlögl, A., Scherer, R., Graimann, B.,

Keinrath, C., Skliris, D., Wörtz, M., Supp, G., Schrank,

C., 2003. Graz-BCI: state of the art and clinical

applications. IEEE Trans. Neural Syst. Rehabil. Eng.

Publ. IEEE Eng. Med. Biol. Soc. 11, 177–180.

doi:10.1109/TNSRE.2003.814454.

Pope, A.T., Bogart, E.H., Bartolome, D.S., 1995.

Biocybernetic system evaluates indices of operator

engagement in automated task. Biol. Psychol., EEG in

Basic and Applied Settings 40, 187–195.

doi:10.1016/0301-0511(95)05116-3.

Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E.,

Delannoy, V., Bertrand, O., Lécuyer, A., 2010.

OpenViBE: an open-source software platform to

design, test, and use brain-computer interfaces in real

and virtual environments. Presence Teleoperators

Virtual Environ. 19, 35–53.

Roberts, R., Callow, N., Hardy, L., Markland, D., Bringer,

J., 2008. Movement imagery ability: development and

assessment of a revised version of the vividness of

movement imagery questionnaire. J. Sport Exerc.

Psychol. 30, 200–221.

Rossini, P.M., Calautti, C., Pauri, F., Baron, J.-C., 2003.

Post-stroke plastic reorganisation in the adult brain.

Lancet Neurol. 2, 493–502.

Shute, V.J., 2008. Focus on Formative Feedback. Rev.

Educ. Res. 78, 153–189. doi:10.3102/003465430731

3795.

Sigrist, R., Rauter, G., Riener, R., Wolf, P., 2013.

Augmented visual, auditory, haptic, and multimodal

feedback in motor learning: a review. Psychon. Bull.

Rev. 20, 21–53. doi:10.3758/s13423-012-0333-8.

Slater, M., 2009. Place illusion and plausibility can lead to

realistic behaviour in immersive virtual environments.

Philos. Trans. R. Soc. Lond. B Biol. Sci. 364, 3549–

3557. doi:10.1098/rstb.2009.0138.

Solhjoo, S., Moradi, M., 2004. Mental task recognition: A

comparison between some of classification methods, in:

BIOSIGNAL 2004 International EURASIP Conference.

pp. 24–26.

Taylor, R.M., II, Hudson, T.C., Seeger, A., Weber, H.,

Juliano, J., Helser, A.T., 2001. VRPN: A Device-

independent, Network-transparent VR Peripheral

System, in: Proceedings of the ACM Symposium on

Virtual Reality Software and Technology, VRST ’01.

ACM, New York, NY, USA, pp. 55–61.

doi:10.1145/505008.505019.

Trompetto, C., Marinelli, L., Mori, L., Pelosin, E., Currà,

A., Molfetta, L., Abbruzzese, G., 2014.

Pathophysiology of Spasticity: Implications for

Neurorehabilitation. BioMed Res. Int. 2014.

doi:10.1155/2014/354906.

Vidaurre, C., Blankertz, B., 2009. Towards a Cure for BCI

Illiteracy. Brain Topogr. 23, 194–198. doi:10.100

7/s10548-009-0121-6.

Vourvopoulos, A., Bermúdez i Badia, S., 2016. Usability

and Cost-effectiveness in Brain-Computer Interaction:

Is it User Throughput or Technology Related?, in:

Proceedings of the 7th Augmented Human

International Conference, AH ’16. ACM, Geneva,

Switzerland. doi:10.1145/2875194.2875244.

Vourvopoulos, A., Cardona, J.E.M., Badia, S.B. i, 2015.

Optimizing Motor Imagery Neurofeedback through the

Use of Multimodal Immersive Virtual Reality and

PhyCS 2016 - 3rd International Conference on Physiological Computing Systems

Motor Priming. Presented at the 2015 International

Conference on Virtual Rehabilitation (ICVR).

Vourvopoulos, A., Faria, A.L., Cameirao, M.S., Bermudez

i Badia, S., 2013. RehabNet: A distributed architecture

for motor and cognitive neuro-rehabilitation, in: 2013

IEEE 15th International Conference on E-Health

Networking, Applications Services (Healthcom).

Presented at the 2013 IEEE 15th International

Conference on e-Health Networking, Applications

Services (Healthcom), pp. 454–459. doi:10.1109/He

althCom.2013.6720719.

Wolpaw, J.R., Birbaumer, N., McFarland, D.J.,

Pfurtscheller, G., Vaughan, T.M., 2002. Brain-

computer interfaces for communication and control.

Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol.

113, 767–791.

Yao, L., Meng, J., Zhang, D., Sheng, X., Zhu, X., 2014.

Combining motor imagery with selective sensation

toward a hybrid-modality BCI. IEEE Trans. Biomed.

Eng. 61, 2304–2312. doi:10.1109/TBME.2013.2287

245.

NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback