Reducing Brain-computer Interaction Training Time with Embodied

Virtual Avatar

Filip

Skola and Fotis Liarokapis

CYENS – Centre of Excellence, Nicosia, Cyprus

Keywords:

Brain-Computer Interfaces, Gamiﬁcation, Motor Imagery, Sense of Agency, Virtual Embodiment, Virtual

Reality.

Abstract:

Brain-computer interfaces (BCI) have been intensely researched to provide a method for controlling comput-

ers, robots, and other machinery using mental activity only. Nevertheless, BCIs remain difﬁcult to use in

everyday life. One of the major BCI paradigms, the motor imagery (MI), showed improved control perfor-

mance when avatar embodiment in virtual reality (VR) was exploited in the BCI system. Control accuracy was

further increased with gamiﬁcation of the MI-BCI training procedure. This paper presents comparative study

of 3 types of MI-BCI training: with the standard protocol, mediated using a virtual avatar, and in a gamiﬁed,

embodied setting with progressive increase of the training pace. Overall analysis of the relationship between

embodiment and BCI performance showed robust embodiment illusion supported by correlation between the

sense of ownership towards the avatar and the sense of agency towards the BCI actions. Interestingly, the

actual control proﬁciency was uncorrelated to the perceived performance and to the sense of ownership. This

could work towards facilitation of the initial training steps similarly to designs exploiting positively biased

feedback.

1 INTRODUCTION

Brain-computer interfaces (BCI) have been subject of

intense research during the last decades, yet there are

numerous issues with their practical applications. Vir-

tual reality (VR) seems to be a promising tool aiding

to bring the BCIs towards users.

VR systems use the body for interaction. Ideally,

any part of the body can take part in the interaction

with an advanced VR system. VR aims to leverage

the most of the sensorimotor loop (Slater, 2009) – the

motor system as the human “output interface” which

is coupled with the sensory system serving for infor-

mation retrieval. VR intervenes in the midst of the

sensorimotor loop, using the data associated to the

motor system (position and orientation of the body

parts) as inputs, providing in turn synthesized per-

cepts with correct transformations to the sensory or-

gans. VR can be seen as a sensory illusion controlled

by bodily movements. This illusion can easily incor-

porate a visual rendering of a selected body (virtual

avatar), acting in synchrony with the user movements.

BCIs aim to allow humans to communicate and

act while bypassing the motor system (Graimann

et al., 2010). One of their main purposes is to enable

communication in people who can not move at all,

which is accomplished by using mental commands

(translated from the brain signals) as the input in-

terface for computers, robotics, and general machin-

ery. Healthy users can leverage BCIs for monitoring

of affective and cognitive states (M

uhl et al., 2014),

e.g. for using information about estimated level of

invested attention as additional input for computer

games or meditation training (Kerous et al., 2017).

This paper presents results on research of brain-

computer communication facilitated using avatar em-

bodiment in VR. The main purpose of combining BCI

and VR was to facilitate the training task for BCI

communication (imagery of hand movements) with

adequate feedback (hand movements performed by

the surrogate body in VR). This research is focused

on the BCI communication paradigm based on mo-

tor imagery (MI, MI-BCIs), which exploits changes

in neural activity generated by consciously attending

to bodily movements.

The aim of this study was to investigate the ad-

vances in user training for MI-BCI usage by leverag-

ing embodiment and gamiﬁcation in VR. Data from

3 groups of participants taken from 2 previous ex-

periments (

Skola and Liarokapis, 2018;

Skola et al.,

Škola, F. and Liarokapis, F.

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar.

DOI: 10.5220/0011049100003124

In Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2022) - Volume 1: GRAPP, pages 7-17

ISBN: 978-989-758-555-5; ISSN: 2184-4321

2019) which were part of a bigger study on embodied

VR MI-BCI training (

Skola, 2020) were studied a) to

investigate the hypothesis that VR embodiment aids

the MI-BCI training procedure (reduces the required

training time), and b) to elucidate the mechanism be-

hind this effect. One studied group of participants was

trained using the standard training method with feed-

back using symbols. Training of the second group

was performed with avatar using virtual embodiment

(using user’s imagined movements as the input and

using the movements of the “surrogate” avatar body

as the output). Finally, the third group was trained

with a gamiﬁed VR application with avatar embodi-

ment. In the simple game wrapped around the train-

ing procedure, participants were trained using a more

visually appealing and engaging environment. More-

over, progressive increase of the training pace was in-

corporated.

This paper presents comparison of the between-

group results, as well as overall analysis of the data on

embodiment and BCI performance. The discussion is

focused on the grounds of the facilitating effect of VR

embodiment to the BCI communication training.

2 BACKGROUND

Control strategy in MI-BCIs consists of focused im-

agery on movement of own hands, feet, etc (Lotte

et al., 2015), mediated by user modulation of the

sensorimotor rhythm in the brain. Although most

people have some sensorimotor rhythm modulation

ability (Dickhaus et al., 2009), training is typically

needed to achieve a reasonable level of control accu-

racy. The training has usually a form of repeated tri-

als where users perform the MI process of speciﬁed

body parts, followed by immediate visual feedback

from the system, allowing users to understand if their

MI effort was recognized successfully. One of the

goals of the training is to enhance the ability to cre-

ate a distinct pattern of neural rhythm change during

the imagined movement, called event-related desyn-

chronization (ERD), in motor cortex parts associated

to the imagined body parts (Kaiser et al., 2014). This

consequently improves the prediction capabilities of

the machine learning side of the BCI system (Lotte

et al., 2015). The training process is a co-adaptation;

while the user is trained with the help of neurofeed-

back, the classiﬁer in the BCI is trained as well (Lotte

et al., 2015).

Training for MI-BCIs is a mentally demanding

process. Users must direct their focused attention

to the movements of hands or feet for prolonged

amounts of time, while no actual movement is per-

mitted during the sessions. Moreover, the commonly

used training protocol uses symbolic visual instruc-

tions and abstract feedback forms to communicate

successfulness of the ongoing training MI commands.

This symbolic visual feedback is shown concurrently

with the MI process, leading to attention split be-

tween comprehension of the feedback and focusing

on movements of the body.

BCI research recently started to highlight the

importance of the human-facing side of the inter-

face (Lotte et al., 2013; Jeunet et al., 2016a; Jeunet

et al., 2016b; Kosmyna and L

ecuyer, 2017; Sollfrank

et al., 2016). Main criticism considered the standard

training protocols, for the reasons they ignore elemen-

tary psychological ﬁndings about the optimal forms

of training (e.g., using progressive or adaptive task

design, exploiting rich feedback modalities (Jeunet

et al., 2016b)). Some of these problems can be allevi-

ated by exploiting gamiﬁcation, which can be deﬁned

as “the use of game design elements in non-game con-

texts” (Deterding et al., 2011). In the BCI context,

gamiﬁcation is helpful especially as it aims to im-

prove immersion and motivation (de Freitas, 2011).

BCIs are often used as game input interfaces, both

for training or other research purposes (Kerous et al.,

2017).

Past research has demonstrated that the visual rep-

resentation of the self in VR can be utilized to facil-

itate the BCI control (Salisbury et al., 2016; Vour-

vopoulos and Berm

udez i Badia, 2016; Vourvopoulos

et al., 2019;

Skola and Liarokapis, 2018;

Skola et al.,

2019). The virtual agent is usually called avatar and

the subjective experience of having and being inside a

virtual body virtual embodiment (Kilteni et al., 2012).

Human brain has special mechanisms for recognition

and self-attribution of the “attached” body, using prior

knowledge and the available sensory data (Jeannerod,

2003). In VR, the visual contact with own body is

cut off and replaced by the rendering of the avatar.

First person view of a body that acts in accordance

with one’s will creates a strong embodiment illusion,

including self-attribution of the avatar body. Conse-

quently, people immersed in VR with an embodied

avatar tend to keep their avatars away from virtual

dangers, eliciting similar physiological responses to

the threats as if they were performed in the physical

reality and the real body was at stake (Meehan et al.,

2002).

When the body is self-attributed, one experi-

ences what is termed the sense of (bodily) owner-

ship (SoO) (Gallagher, 2000). The sense of being

the author (agent) of the voluntary actions is termed

the sense of agency (SoA). The SoA can be deﬁned

in terms of the authorship of the voluntary move-

VISIGRAPP 2022 - 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

Figure 1: Openvibe implementation of the Graz training

protocol (symbolic guidance); top: instruction to perform

a right hand MI trial, bottom: feedback (an extending bar)

representing a relatively conﬁdent classiﬁer decision that

the current participant’s mental effort belongs to the right

hand MI class.

ments (Blanke and Metzinger, 2009), but more com-

monly the deﬁnition includes also the covert actions

such as creating an intent or a thought in the stream of

thoughts (Gallagher, 2000; Gallagher, 2007). BCIs al-

low manifestation of one’s SoA using covert actions;

i.e., BCIs allow for translation from the intention to

an action without any movement.

3 METHODS

3.1 Compared Variables

Embodiment was quantiﬁed using questionnaires on

SoO and SoA (standard questions from studies on

body ownership were used, based on (Botvinick and

Cohen, 1998; Longo et al., 2008)), answered on a Lik-

ert scale from -3 to +3, after the experiment. BCI per-

formance was calculated as a) total time spent in the

correct MI state (recognized by the classiﬁer) and b)

percentage of successfully recognized MI actions (a

normalization between study (

Skola and Liarokapis,

2018) in Section 3.2 and the study (

Skola et al., 2019)

in Section3.3 had to be performed, the details on nor-

malization are provided in Section 4.1). Addition-

ally, bit-transfer rate (BTR) was calculated and used

in the comparison. For the purposes of comparing the

ﬁrst run of training (when the feature set for the ini-

tial training of the classiﬁer was being created), cross-

validation classiﬁcation accuracy (CCA) obtained and

analyzed.

Figure 2: Screenshot from the VR scene used for the train-

ing in the Embodied group of participants (resting phase).

3.2 Study Comparing Standard and

Embodied Training

The ﬁrst study leveraging embodied VR training for

MI-BCI (

Skola and Liarokapis, 2018) aimed to assess

the difference between usage of the standard Graz MI-

BCI training protocol with symbolic feedback (see

Figure 1) and the newly designed embodied paradigm

for BCI training (Figure 2). For purposes of creat-

ing the embodied training, a body ownership illusion

with binding of the SoO and the SoA using actions

performed with the MI-BCI was necessary to be im-

plemented. That was achieved by initiating the ex-

periment directly with the synchrony between mental

effort and visual outcomes of the VR environment, to

facilitate the body ownership transfer (embodiment)

from the beginning of the experiment. In other words,

active thought process focused on the imagery hand

movements bound to the visual representation of the

moving virtual hands was the driving mechanism be-

hind BCI-mediated embodiment.

Design of the VR scenes for interaction with MI-

BCIs is subject to constraints, arising from the neces-

sity to maintain a stable bodily posture without en-

gagement of voluntary muscles, including minimiza-

tion of the eye movements. The reasons for that are

that the ongoing EEG signals are prone to artifacts,

generated either by both bodily and external sources.

Bodily artifacts are mainly caused by muscle activ-

ity from the facial area and the muscles around the

neck. But even movement of the other parts of the

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar

body would have unwanted effects on the signal, ei-

ther by generation of the EEG artifacts, or by contam-

ination of the signals with brain signals originating

from the activations in the motor cortex.

In the very ﬁrst run of the training, the EEG

data for personalized per-trial neurofeedback were not

available. For that reason, movements of the avatar

were carried out without an input from the BCI (the

avatar performed the hand movements with natural

speed). Still, participants were required to synchro-

nize their mental imagery to the observed movements,

to a) provide an initial feature set for the classiﬁer

training (system training with the features containing

MI patterns); and b) facilitate the embodiment illu-

sion due to the synchrony between the users’ MI effort

and the visual feedback. Motor action observation is

known to contribute to ERD strengthening during the

MI (Kondo et al., 2015), further facilitating the initial

training step.

The proposed embodied training aimed to cor-

rect sub-optimal elements of the training procedure,

especially the feedback modality (incomprehensive

guidance causing split between the task and the pro-

vided feedback), touching the motivational aspects of

the training. The main feedback method was using

the movements of the avatar, speciﬁcally its speed;

natural movements indicated good BCI performance,

while movements that were slowed-down (eventually

almost to the point of stopping) indicated problems of

the BCI system to comprehend the user EEG inputs.

From the 30 participants in this study, control

group (N = 15) was trained using the Graz training

protocol (Openvibe implementation) on the standard

computer screen, while experimental group (N = 15)

was trained using the novel embodied protocol. BCI

control was mediated using imagined movement of

left and right hand, aiming to push the virtual but-

ton in front of the participants. After two runs of the

training, all participants accomplished the same task –

pushing of the virtual button. Brain signals were col-

lected using a 20-sensor set-up based on a lightweight

wireless EEG device.

Questionnaire results on embodiment revealed the

mean SoO was equal to 0.700 (SD = 1.670) and the

mean SoA rating was 1.400 (SD = 1.283). These val-

ues show a relatively high sense of embodiment for

the novel design. See the original paper (

Skola and

Liarokapis, 2018) for a detailed information on the

study.

Figure 3: VR feedback representing a recently ﬁnished right

hand MI trial in the Progressive group.

3.3 Study using Gamiﬁcation and

Progressive Pace for Training

The novel VR application created as a gamiﬁed vari-

ant of the MI-BCI training (

Skola et al., 2019) kept

most from the standard trial-based form of the MI-

BCI training (randomized training trials grouped into

several stages, with given cue before each trial, and

randomized rest period after each trial). Gamiﬁed

training employed themed environment and score

points, while the challenge in the game arose mainly

from the progressive increase of the speed across sev-

eral training runs, or levels (categories of motivational

affordances were adapted from (Hamari et al., 2014)).

Feedback on user progress was provided using three

modalities:

1. Embodied feedback mediated by the speed of

avatar hand movements (real-time)

2. “Score” displayed in the VR scene (post-trial

quantiﬁcation of the trial accuracy)

3. Vibrotactile feedback to the corresponding hand

mediated by the VR controllers (real-time)

The VR scene was set in the outer space, the par-

ticipant was virtually transferred inside a cockpit of

a spaceship with an Earth-like planet ahead (see Fig-

ure 3). The gamiﬁed training objectives consisted of

shooting ﬂying asteroids depending on their position

(using left hand MI if the asteroid ﬂew from the left

side of the spaceship, right hand MI if from the right

side). The spaceship contained a simple control panel

consisting of a low number of interaction elements

that triggered the weapons shooting the asteroids ﬂy-

ing towards the planet.

Training was designed as progressive; i.e., pace of

each run was higher than the preceding one. The user

evaluation consisted of six runs of the training, while

the ﬁrst run consisted of MI facilitated by motor ob-

servation and the other runs provided participant with

VISIGRAPP 2022 - 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

increasingly faster paced training. Last run inquired

into a modality change experiment (post-trial discrete

feedback only, without the real-time feedback). For

more details please see the paper (

Skola et al., 2019).

The VR training environment was evaluated in

a between-subject user study with 19 participants

(N = 20 including a participant falling below chance

level) performing 6 runs of the training. Signals were

collected with 28 sensors using the same device as

in the previous study. Questionnaires revealed a sur-

prisingly high positive affect after the experimental

session (mean 6.763, SD = 0.348, on a scale from 1

to 7). No participant reported the engagement or in-

terest lower than 6 out of 7 points. Mean of the SoO

statements was 0.768 (SD = 1.190) and the mean SoA

was 1.290 (SD = 0.947).

3.4 Compared Groups and Results

The following 3 groups are analyzed in this paper:

• From the user study comparing the embodied

VR training to the standard method (

Skola and

Liarokapis, 2018)

– Control group (participants trained with the

standard protocol)

– Embodied group (participants trained with em-

bodied avatar in VR)

• From the user study on gamiﬁed progressive train-

ing in embodied VR (

Skola et al., 2019)

– Progressive/Gamiﬁed group (participants

trained with embodied avatar in gamiﬁed,

progressive VR)

To perform the analysis covering all these

datasets, the following was taken into account:

• Accuracy and BTR in the 3rd run (maximal num-

ber of common training runs for all datasets)

• Accuracy and BTR of the best run per participant

• Number of participants not surpassing the chance

level

• CCA in the ﬁrst run without real-time feedback

(to study the inﬂuence of the initial VR embodi-

ment with motor action observation)

Non-parametric statistical tests were used due to

non-normal distribution of the data.

4 RESULTS

4.1 Normalization of the Performance

Results

4.1.1 Accuracy Metric Used in the Analyses

In (

Skola and Liarokapis, 2018), the on-line accuracy

was reported as an average of percentages of time

in each trial spent in the correct MI state. On-line

accuracy metric called “hit-wise accuracy” (percent-

age of trials with >50% successful time) was used

in (

Skola et al., 2019), representing the percentage of

successfully issued commands using a two-class BCI.

In comparison to the accuracy metric from (

Skola and

Liarokapis, 2018), it also shows a stronger correlation

to the CCA. In Control group r = 0.746, p = 0.543 ver-

sus r = 0.523, p = 0.229; in Embodied group r = 0.619,

p = 0.024 versus r = 0.480, p = 0.097; in Progressive

group r = 0.714, p = 0.001 versus r = 0.620, p = 0.005

(Spearman tests). Consequently, the “hit-wise accu-

racy” was utilized for the overall analysis, and all

the performance results from (

Skola and Liarokapis,

2018) were recalculated for purposes of this paper.

4.1.2 Differences between the Accuracy Metrics

and the BTR

In (

Skola and Liarokapis, 2018), the average trial

length was 14.805 s (feedback training) and 14.628 s

(evaluation run), including the rest periods. The tri-

als were signiﬁcantly shortened in the follow-up study

(min = 6.098 s, max = 12.853 s), see Table 2. Con-

sequently, the former BTR values are very low, and

the on-line accuracy serves better for comparison of

the performance in the three conditions. In particu-

lar, accuracy in the 3rd run can be used to compare

the performance after the same amount of training tri-

als (from the two preceding training runs, with one

including feedback).

Nonetheless, the average of the total training du-

ration was 24.616 minutes in the comparison study

and 28.153 minutes in the gamiﬁed progressive study.

The best run accuracy represents the performance af-

ter a comparable time spent training (rather than after

the same number of training trials performed). But

still, the length of the training in the latter study was

increased by 14.369%.

The BTR values demonstrate the performance in-

crease obtained by employment of the higher pace and

the progressive design of the Progressive group, com-

pared to the initial embodied design of the Embodied

group.

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar

Table 1: Comparison of the BCI performance across all tested groups; average on-line accuracy, BTR, and SD in parentheses.

Groups marked as w/s-ch-l include sub-chance-level participants.

Group N Accuracy (run 3) Accuracy (best run) BTR (run 3) BTR (best run)

Control 7 62.286% (8.655) 62.286% (8.655) 0.267 (0.396) 0.267 (0.396)

Embodied 13 65.385% (16.132) 67.153% (14.960) 0.700 (1.306) 0.728 (1.292)

Progressive 19 67.105% (10.603) 75.842% (11.251) 0.825 (0.949) 1.992 (1.992)

Control (w/s-ch-l) 15 53.067 % (11.386) 53.067% (11.386) 0.160 (0.288) 0.160 (0.288)

Embodied (w/s-ch-l) 15 63.333% (15.886) 64.867% (15.109) 0.606 (1.234) 0.631 (1.223)

Progressive (w/s-ch-l) 20 66.250% (11.007) 74.400% (12.709) 0.784 (0.942) 1.893 (1.606)

Table 2: Averaged duration of a trial including the rest pe-

riod in each of the runs (levels) in the Progressive group

(Avg. trial length) and a maximal duration of the MI part

(user effort) of a trial (Max. MI length).

Run # Avg. trial length Max. MI length

2nd 12.853 8.8

3rd 8.977 5.15

4th 7.143 3.93

5th 6.098 3.325

6th 7.158 3.9

4.2 BCI Performance Comparison

4.2.1 Accuracy and BTR per Training Method

Table 1 contains average performance results per-

group (accuracy and BTR), while Figure 4 shows box-

plots representing the average accuracy. The method

used in the Progressive group produced the best re-

sults in all performance metrics. The analyses were

focused on the subsets of participants who were able

to surpass the chance levels, but results including the

sub-chance-level participants (N = 11, details in Ta-

ble 3) are stated as well.

Performance variability among BCI users ac-

counted for a low statistical signiﬁcance of the dif-

ferences among the groups. Generally, the differ-

ences are worth testing for statistical signiﬁcance if

the sub-chance-level participants are included in the

sample. Sample sizes of groups after pruning of the

subjects without any control become low and strongly

●

Control (run 3) Embodied (run 3) Progressive (run 3) Progressive (best run)

50 60 70 80 90 100

Group

Accuracy (%)

Figure 4: Boxplots showing per-group BCI performance

(accuracy).

unbalanced. The strongest difference (tested with

Wilcoxon test) is between accuracy values of Progres-

sive and Control group with W = 51.500 and p = 0.001

(with the sub-chance level participants included).

4.2.2 Participants below the Chance Level

Performance was analyzed from the subset of partici-

pants surpassing the 50% chance level in at least one

of the runs. Nevertheless, participants below that line

are also important to take into account, as they can

be considered an indicator of the BCI illiteracy (ra-

tio of untrainable participants). Table 3 contains an

overview of the percentage of participants not able to

surpass the chance level in any of the runs in a session.

Table 3: Number of participants (s-ch-l stands for sub-

chance-level) not surpassing chance level (third column) in

any of the runs per MI-BCI group.

Group N N [s-ch-l] Rate

Control 15 8 56.333%

Embodied 15 2 13.333%

Progressive 20 1 5.000%

Even though the comparison was carried out

among groups with different number of runs (the Pro-

gressive group with 5 on-line runs and the other two

groups with 2 on-line runs), all but the one partici-

pant in the Progressive achieved >50% accuracy al-

ready in the ﬁrst two runs. Thus even after compar-

ison of the data from the ﬁrst two runs only, the ﬁg-

ures remain the same. This might be caused by sim-

ply more engaging design of the Progressive training

compared to the Embodied training, but the number

of sub-chance-level individuals is not very different

in these two groups. Nevertheless, comparison to the

Control group indicated both embodied designs as a

signiﬁcant improvement.

4.3 Embodiment and BCI Performance

4.3.1 Effect of the Initial Embodiment

Effect of the combined MI and motor observation dur-

ing the ﬁrst stage of the training was assessed using

VISIGRAPP 2022 - 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

●

Control Embodied Progressive

60 65 70 75 80 85

Group

CA (%)

Figure 5: Boxplots showing CCA of the dataset correspond-

ing to the initial training. The results are shown per training

method, without the sub-chance-level participants.

the CCA in the ﬁrst run (see Figure 5). This CCA

represents the initial ability of the classiﬁer to distin-

guish between the classes based only on the feedback-

less training (VR avatar movements were shown, but

with no input from the BCI), thus allowing to estimate

how much the embodiment facilitated the differences

between participants’ ERDs for the left and right hand

MI.

Results show that the Control group had a lower

CCA (mean 66.267%, SD = 4.621) than the other

groups (Embodied mean 68.600%, SD = 7.089; Pro-

gressive mean 70.821%, SD = 5.268). The difference

is statistically signiﬁcant between the Progressive and

the Control groups (W = 222, p = 0.017), suggesting

that the ﬁrst stage of training with embodiment illu-

sion indeed helped the initial classiﬁer training, com-

pared to the training with the standard Graz protocol.

Future work should investigate the effect of move-

ment observation (in comparison to the intentional

MI) on the initial ERDs. Removal of the real-time

feedback in the progressive MI-BCI training study

provides evidence that the training is not completely

hindered after the movement observation is removed

from the feedback, but more data on the relationship

between MI, motor observation, and ERD generation

should be gathered.

4.3.2 SoO, SoA, and BCI Performance

First of all, Progressive group had the SoO correlated

to the magnitude of the ERD. High correlation coefﬁ-

cient and signiﬁcance were reached in this group (r = -

0.698, p = 0.001, N = 18; see Figure 6). But in the

other two groups, the ERD–SoO relationship was not

conﬁrmed.

Very strong correlation (r = 0.581, p = 0.000,

N = 49) was present also between the reported SoO

and the reported SoA (see Figure 7). Interestingly,

Figure 6: Relationship between the SoO and the ERD

in (

Skola et al., 2019) (Embodied group). Apart from

an outlying participant, the higher SoO was bound to the

stronger ERD.

●

● ●

●

● ●

●

● ●

●

●●

●

●●

●

−2 −1 0 1 2 3

−3 −2 −1 0 1 2 3

SoA

SoO

Figure 7: Scatterplot showing the relationship between the

reported SoO and the reported SoA (Spearman r = 0.581,

p = 0.000, N = 49). Participants tended to experience a high

ownership towards the virtual hands controlled by the MI-

BCI when the notion of control was also high, indicating

sense of embodiment towards the virtual avatar.

the SoA and the BCI performance were not correlated

(r = 0.064, p = 0.660, N = 49), see details on Figure 8.

Finally, correlation between the SoO and the on-

line performance was not found (the correlation co-

efﬁcient was with r = 0.018, p = 0.901, N = 49; see

Figure 9).

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar

●

● ●●

● ●

●

● ●

●

● ●

●

40 50 60 70 80 90 100

−2 −1 0 1 2 3

On−line accuracy of the best run (%)

SoA

Figure 8: Relationship between the SoA and the actual con-

trol proﬁciency (on-line accuracy of the best run). As the

hand movements of the avatar correlated to the participant

intentions in case of a good BCI performance (and partici-

pants were aware of this fact), correlation between the BCI

performance and the SoA (as the self-report on the per-

ceived performance) was expected. However, this was not

conﬁrmed.

●

●●

●

●●

●

●●

●

● ●

●

●●

●

40 50 60 70 80 90 100

−3 −2 −1 0 1 2 3

On−line accuracy of the best run (%)

SoO

Figure 9: The on-line BCI performance (accuracy) was not

related to the perceived SoO in the session.

5 DISCUSSION

Originally, we expected to see a linear relationship be-

tween the SoO and on-line BCI performance. The

reason for that was that the SoA towards own actions

is among the primary mechanisms that bind the SoO

for body parts (the belief that one controls a body

leads to the belief it is his/her body). Secondly, the

embodiment, manifested by the SoO towards avatar,

was hypothesized to facilitate the training, i.e., to al-

low a better BCI performance. In other words, a low

performance should hinder the body ownership trans-

fer, and a good performance should enable it; a high

level of the body ownership transfer should facilitate

the training, and a low level should hinder it.

Instead, the SoO was bound together with the

SoA. Importantly, if the SoO indeed depended on the

BCI performance, a standard novice BCI user start-

ing at a low level of performance could not leverage

the hypothesized function of the embodiment in train-

ing to actually improve (a low SoO during a poor BCI

control level would be hard to increase if the higher

SoO required better BCI control and vice versa). It

is thus beneﬁcial that the level of the SoO did not

rely as much on the performance as it did on the sub-

jectively perceived performance; i.e., the perceived

SoA towards the BCI actions (see Figure 7). Thanks

to this relationship, the BCI-mediated embodiment

arose even in poorly-performing participants.

Our ﬁnding that the binding mechanism of the

body ownership illusion was independent on con-

trol proﬁciency helps to clarify why the embodied

VR yielded better performance results. Biased forms

of feedback in MI-BCI training (feedback indicating

better performance than the actual performance) have

been utilized to facilitate the training process (Bar-

bero and Grosse-Wentrup, 2010; Faller et al., 2012).

Usage of embodiment seems to help participants in

tackling the difﬁculties of the ﬁrst training stages with

a similar mechanism.

Whether the gap between the actual and the per-

ceived performance gets eventually smaller and the

users naturally sharpen their skills in self-assessment

based on the feedback mapped onto the VR body, or

not, needs to be studied in a longitudinal study. It

is a question closely related to why the SoA was un-

correlated to the actual performance (accuracy). It is

possible that the interpersonal differences in the locus

of control and the self-discipline had major inﬂuence

to the perceived SoA. These effects can be even more

pronounced when only a single exposure to the BCI

system is made, and the self-assessment would im-

prove after repeated exposure (when comparison with

past performances can be done).

Finally, correlation between the reported SoO and

the strength of the ERD in MI trials provides fur-

ther evidence in favor of the hypothesis that train-

ing with avatar embodiment facilitates the MI-BCI

training by inducing the sense of embodiment. How-

ever, this effect was not consistent throughout the en-

tire dataset and was rather observed only in the Pro-

gressive group. This can be due to the largest num-

ber of trials and the highest achieved performances

VISIGRAPP 2022 - 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

(compared to the other groups), but further valida-

tion of this effect is needed. In case a link between

the SoO and a correlate of the MI is conﬁrmed in a

larger study, it will provide more evidence that subjec-

tively experiencing the avatar embodiment facilitates

pre-requisites for successful MI-BCI training (such as

generation of the stronger ERDs).

5.1 Limitations

Low and varying number of participants in the ana-

lyzed datasets is certainly the main limitation of this

study. This limits the extent to which generalizations,

especially from the between-group comparisons, can

be made and requests for more studies examining the

outlined phenomena with greater number of subjects.

Comparisons between the groups are further weak-

ened due to differences between the methodology of

the studies analyzed. Despite our effort to compensate

this issue with normalization of the data and providing

explanations on differences, it must be noted that data

from two, albeit closely related, studies were used in

this paper.

6 CONCLUSIONS

With MI-BCIs, people can control devices using an

interface that bypasses the motor functions of the

body, but exploits them at the same time. Even if only

at the imagery level, movement is still exploited in

this BCI paradigm, and motor actions must be under-

stood and well-imitated covertly to convey the control

signals in the EEG representations. The problematic

case when control is hindered due to an insufﬁcient

level of the imagery skills can be mitigated with the

embodied feedback.

Providing a guidance could sound like a poor rea-

son to create an MI-BCI completely enclosed within

the VR, but it is important to take into account the

target user group, currently composed of people with

paralysis or disabilities. Embodied MI-BCI training

in VR could help the users similarly to the VR-based

BCI-assisted rehabilitation – by helping to reconnect

with the bodily functions.

Gamiﬁcation and progressive increase of the train-

ing pace were exploited for the goals of boosting

user attention and motivation, resulting in further in-

crease of the BCI performance compared to VR alone.

Moreover, the number of participants not making any

progress in one session was lower in comparison to

the standard protocol, and participants reported a high

positive affect after the end of the session. Working

with the user motivation and affect in general may be

the key to the future of BCI research, as the perfor-

mance results are inseparable from the effort invested

by the participants.

The conditions enabling the body ownership

transfer in the BCI control were much looser in com-

parison to the cases when motor control is used to

convey one’s will. The reports on the SoA towards the

actions performed by the surrogate body were rather

high, and they were strongly correlated to the reported

SoO towards the virtual agent, suggesting a robust

sense of embodiment during the MI-BCI training. At

the same time, actual performance in BCI actions did

not go hand in hand with the perceived performance.

As the SoO was also strongly correlated to the mag-

nitude of the average ERD in the latter BCI experi-

ment, it seems that employment of the embodiment

could facilitate the training by a combination of the

elevated SoA (similarly as in the training designs us-

ing a positively biased feedback) and the strengthened

ERDs during MI when the avatar body is temporarily

accepted as the own body.

7 FUTURE WORK

The results provided in this work demonstrated that

the ﬁrst-time and inexperienced users beneﬁt from

the embodied training. The research was limited by

the short overall duration of the training procedure,

and it should be further researched if the VR train-

ing is feasible for longer periods of time. It is likely

that the several technological limitations concerning

the VR equipment (e.g., wearing the uncomfortable

HMD, practical problems due to the concurrent usage

of an HMD and the EEG) will be overcome due to

the technological advances accompanying the popu-

larization of VR technology. This would increase the

chances of a less demanding VR-BCI training in the

near future. If the training with HMDs could be on a

level of comfort comparable to the standard computer

screens, the conceptual advantage of the VR MI-BCI

could easily prevail.

Nonetheless, even if the whole training procedure

was mediated using VR, the typical BCI user needs to

be sufﬁciently accurate in control of the system even

outside VR, for many practical reasons. To make this

transition easier, AR seems to be the perfect candidate

for the future work. Embodiment in AR is by far not

as much explored as the VR embodiment, mainly be-

cause AR can easily incorporate the existing body of

the user. In the BCI-mediated communication, where

users are typically not moving at all, avatar in AR

makes perfect sense.

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar

ACKNOWLEDGMENTS

This research was partially supported by the project

that has received funding from the European Union’s

Horizon 2020 Research and Innovation Programme

under Grant Agreement No 739578 and the Govern-

ment of the Republic of Cyprus through the Deputy

Ministry of Research, Innovation and Digital Policy.

REFERENCES

Barbero, l. and Grosse-Wentrup, M. (2010). Biased feed-

back in brain-computer interfaces. Journal of Neuro-

Engineering and Rehabilitation, 7(1):34.

Blanke, O. and Metzinger, T. (2009). Full-body illusions

and minimal phenomenal selfhood. Trends in Cogni-

tive Sciences, 13(1):7–13.

Botvinick, M. and Cohen, J. (1998). Rubber hands’

feel’touch that eyes see. Nature, 391(6669):756.

de Freitas, S. (2011). Technology: Game for change. Na-

ture, 470:330–331.

Deterding, S., Dixon, D., Khaled, R., and Nacke, L. (2011).

From game design elements to gamefulness: deﬁn-

ing” gamiﬁcation”. In Proceedings of the 15th inter-

national academic MindTrek conference: Envisioning

future media environments, pages 9–15.

Dickhaus, T., Sannelli, C., M

uller, K.-R., Curio, G., and

Blankertz, B. (2009). Predicting BCI performance to

study BCI illiteracy. BMC Neuroscience, 10(Suppl

1):P84.

Faller, J., Vidaurre, C., Solis-Escalante, T., Neuper, C., and

Scherer, R. (2012). Autocalibration and Recurrent

Adaptation: Towards a Plug and Play Online ERD-

BCI. IEEE Transactions on Neural Systems and Re-

habilitation Engineering, 20(3):313–319.

Gallagher, S. (2000). Philosophical conceptions of the self:

implications for cognitive science. Trends in Cogni-

tive Sciences, 4(1):14–21.

Gallagher, S. (2007). The Natural Philosophy of Agency.

Philosophy Compass, 2(2):347–357.

Graimann, B., Allison, B., and Pfurtscheller, G., editors

(2010). Brain-computer interfaces: revolutionizing

human-computer interaction. Frontiers collection.

Springer, Heidelberg. OCLC: ocn707710772.

Hamari, J., Koivisto, J., and Sarsa, H. (2014). Does Gam-

iﬁcation Work? – A Literature Review of Empirical

Studies on Gamiﬁcation. In 2014 47th Hawaii Inter-

national Conference on System Sciences, pages 3025–

3034.

Jeannerod, M. (2003). The mechanism of self-recognition

in humans. Behavioural brain research, 142(1):1–15.

Jeunet, C., Jahanpour, E., and Lotte, F. (2016a). Why stan-

dard brain-computer interface (BCI) training proto-

cols should be changed: an experimental study. Jour-

nal of neural engineering, 13(3):036024.

Jeunet, C., N’Kaoua, B., and Lotte, F. (2016b). Advances in

user-training for mental-imagery-based BCI control:

Psychological and cognitive factors and their neural

correlates. Progress in brain research, 228:3–35.

Kaiser, V., Bauernfeind, G., Kreilinger, A., Kaufmann, T.,

ubler, A., Neuper, C., and M

uller-Putz, G. R. (2014).

Cortical effects of user training in a motor imagery

based brain–computer interface measured by fNIRS

and EEG. NeuroImage, 85:432–444.

Kerous, B., Skola, F., and Liarokapis, F. (2017). EEG-based

BCI and video games: a progress report. Virtual Re-

ality.

Kilteni, K., Groten, R., and Slater, M. (2012). The Sense of

Embodiment in Virtual Reality. Presence, 21(4):373–

387.

Skola, F. and Liarokapis, F. (2018). Embodied VR environ-

ment facilitates motor imagery brain–computer inter-

face training. Computers & Graphics, 75:59–71.

Skola, F., Tinkov

a, S., and Liarokapis, F. (2019). Progres-

sive Training for Motor Imagery Brain-Computer In-

terfaces Using Gamiﬁcation and Virtual Reality Em-

bodiment. Frontiers in Human Neuroscience, 13:329.

Kondo, T., Saeki, M., Hayashi, Y., Nakayashiki, K., and

Takata, Y. (2015). Effect of instructive visual stimuli

on neurofeedback training for motor imagery-based

brain-computer interface. Human Movement Science,

43:239–249.

Kosmyna, N. and L

ecuyer, A. (2017). Designing Guiding

Systems for Brain-Computer Interfaces. Frontiers in

Human Neuroscience, 11.

Longo, M. R., Sch

ur, F., Kammers, M. P. M., Tsakiris,

M., and Haggard, P. (2008). What is embodiment? A

psychometric approach. Cognition, 107(3):978–998.

Lotte, F., Bougrain, L., and Clerc, M. (2015). Electroen-

cephalography (EEG)-Based Brain–Computer Inter-

faces. Wiley Encyclopedia of Electrical and Electron-

ics Engineering.

Lotte, F., Larrue, F., and Muehl, C. (2013). Flaws in cur-

rent human training protocols for spontaneous Brain-

Computer Interfaces: lessons learned from instruc-

tional design. Frontiers in Human Neuroscience,

7:568.

Meehan, M., Insko, B., Whitton, M., and Brooks, Jr., F. P.

(2002). Physiological Measures of Presence in Stress-

ful Virtual Environments. In Proceedings of the 29th

Annual Conference on Computer Graphics and Inter-

active Techniques, SIGGRAPH ’02, pages 645–652,

New York, NY, USA. ACM.

uhl, C., Allison, B., Nijholt, A., and Chanel, G. (2014). A

survey of affective brain computer interfaces: princi-

ples, state-of-the-art, and challenges. Brain-Computer

Interfaces, 1(2):66–84.

Salisbury, D. B., Dahdah, M., Driver, S., Parsons, T. D.,

and Richter, K. M. (2016). Virtual reality and brain

computer interface in neurorehabilitation. Proceed-

ings (Baylor University. Medical Center), 29(2):124–

127.

Skola, F. (2020). Reducing the motor imagery brain-

computer interface training time with embodiment in

virtual reality.

Slater, M. (2009). Place illusion and plausibility can lead to

realistic behaviour in immersive virtual environments.

VISIGRAPP 2022 - 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications

Philosophical Transactions of the Royal Society B: Bi-

ological Sciences, 364(1535):3549–3557.

Sollfrank, T., Ramsay, A., Perdikis, S., Williamson, J.,

Murray-Smith, R., Leeb, R., Mill

an, J., and K

ubler, A.

(2016). The effect of multimodal and enriched feed-

back on SMR-BCI performance. Clinical Neurophys-

iology, 127(1):490–498.

Vourvopoulos, A. and Berm

udez i Badia, S. (2016). Motor

priming in virtual reality can augment motor-imagery

training efﬁcacy in restorative brain-computer interac-

tion: a within-subject analysis. Journal of NeuroEngi-

neering and Rehabilitation, 13:69.

Vourvopoulos, A., Pardo, O. M., Lefebvre, S., Neureither,

M., Saldana, D., Jahng, E., and Liew, S.-L. (2019). Ef-

fects of a Brain-Computer Interface With Virtual Re-

ality (VR) Neurofeedback: A Pilot Study in Chronic

Stroke Patients. Frontiers in Human Neuroscience, 13.

Reducing Brain-computer Interaction Training Time with Embodied Virtual Avatar