Brain Computer Interface and Eye-tracking for

Neuropsychological Assessment of Executive Functions:

A Pilot Study

Pietro Cipresso

1,*

, Paolo Meriggi

, Laura Carelli

, Federica Solca

, Barbara Poletti

Dorothée Lulé

, Albert C. Ludolph

,Vincenzo Silani

and Giuseppe Riva

Applied Technology for Neuro-Psychology Lab, IRCCS Istituto Auxologico Italiano

Via G. Pellizza da Volpedo, 41, 20149, Milano, Italy

Polo Tecnologico – Biomedical Technology Department

Fondazione Don Carlo Gnocchi Onlus, Via Capecelatro, 66, 20148, Milano, Italy

Department of Neurology and Laboratory of Neuroscience, “Dino Ferrari” Center

Università degli Studi di Milano, IRCCS Istituto Auxologico Italiano

Piazzale Brescia, 20, 20149, Milano, Italy

Department of Neurology, University of Ulm, Oberer Eselsberg 45, 89081, Ulm, Germany

Abstract. In this study we explored the use of Brain Computer Interface (BCI)

and Eye-Tracking (ET) technology both as augmentative and alternative

communication (AAC) tool and to assess cognitive deficits. Specifically, we

focused on the possible development of a neuropsychological battery for

cognitive assessment based on the integration of BCI and ET tools. To

preliminary test this approach we assessed eight healthy subjects with a

widespread used cognitive task. AAC and usability of both instruments have

also been evaluated with the aim to fine-tune the overall system architecture for

clinical use.

1 Introduction

Some of the most consistently reported cognitive changes regards frontal executive

functions, e.g. verbal fluency, attention, working memory, planning and abstract

reasoning [1-6]. However, the assessment of cognitive impairment still remains a

problematic issue in patients, because of the possible presence of severe physical

disabilities, including movement impairment, paralysis in the advanced stages and

dysarthria, which interfere with the outcome of traditional neuropsychological testing.

New technologies to enable communication have been recently used in several

studies; however, a comprehensive battery for cognitive assessment has never been

implemented with these promising methodologies. Among these methods, Brain

Computer Interface (BCI) and Eye Tracking (ET) are the most promising

Cipresso P., Meriggi P., Carelli L., Solca F., Poletti B., Lulé D., C. Ludolph A., Silani V. and Riva G..

Brain Computer Interface and Eye-tracking for Neuropsychological Assessment of Executive Functions: A Pilot Study.

DOI: 10.5220/0003893100790088

In Proceedings of the 2nd International Workshop on Computing Paradigms for Mental Health (MindCare-2012), pages 79-88

ISBN: 978-989-8425-92-8

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

technologies. BCI uses neurophysiological signals as input commands to control

external devices, while ET allows the measurement of eye position and movements.

To date, no applications have been developed, using ET as a communication

device in order to administer traditional cognitive tasks to patients.

The main disadvantage in the use of ET systems is that they require good ocular

mobility, and the absence of important visual deficits; the former may be lost or

altered in the final stages of disease, and the latter may be present in patients of

advanced age, thus forbidding the use of this device. In fact, ET may not be

proficiently used in case of poor or lack of eye-motor control, such as in late stages of

the disease. In this case there is the need of a more direct interface between voluntary

cortex activity and the computer. BCI may offer an interesting answer to this issue

with a growing number of different paradigms proposed. The most frequently used is

the P300, an event related potential (ERP) elicited by infrequent task stimuli, that

occurs 200-700 ms; it is typically recorded over central-parietal scalp locations [8-10]

and it can also be used by patients suffering from complete paralysis and impairment

in oculo-motor dysfunctions, such as locked-in patients.

It is notable, however, that 20% of subjects are not proficient in using BCI; this

phenomenon is called “BCI illiteracy” [11] and it is due to the fact that some users do

not produce brain activity detectable at the scalp level, independently from the health

conditions: even about 10% of healthy subjects do not produce “usable” P300.

With regard, as an example, to Amyotrophic lateral sclerosis (ALS) patients,

studies have shown that some of them produce less typical ERPs than healthy

matched subjects [12]; [13]. A previous ERP study in patients with sporadic ALS

found that P3a and P3b amplitudes of ALS patients were lower and P3a latencies

were significantly longer compared with the controls [14]; ERP recordings in non-

demented patients with sporadic ALS also showed prolonged N200 and P300

latencies compared to healthy controls [15]. Ogawa and colleagues [16], by

employing neuropsychological measures, event-related potentials (ERPs) and clinical

scales, studied a sample of patients with early-stage sporadic ALS. They found that

patients with the bulbar-onset type showed marked prolongation of P3 latency

compared to patients with the limb-onset type and controls. Furthermore, bulbar

functional rating scale correlated with prolonged P3 latency and low P3 amplitude.

Additionally, patients with bulbar-onset ALS had consistently poorer cognitive test

performance than those with limb-onset ALS [17]. These results represent a challenge

for the use of P300 as an input signal in BCIs. Kübler and Birbaumer [7] investigated

the relationship between the level of motor and physical impairment and the ability to

use brain computer interface by comparing three different BCI systems (P300, SCP

and SMRs, i.e. sensorimotor rhythms). They found no continuous decrement in BCI

performance with physical decline, even in the completed locked-in state (CLIS)

where no communication was possible.

Two important criteria in order to evaluate the feasibility of a BCI system are

speed and accuracy [18]. The former is related to the fact that the more rapidly a BCI

can be controlled, the greater quantity of information can be produced by the user and

the greater the chance for effective communication. Obviously, compared to verbal

speech production, communication rate is severely reduced with BCI. With regard to

accuracy, it consists of the percentage of correct selections per time interval. A wrong

selection could turn into an error in communication, with both practical and

psychological consequences for the user. In order to avoid this, the BCI system must

be equipped with options that allow the user to correct wrong selections. A balance

between speed and accuracy should be identified.

A recently funded project, “eBrain: BCI-ET for ALS", [19] aims to evaluate BCI

P300 technique and eye-tracking technology both as AAC systems and as cognitive

assessment tool with ALS patients.

Before beginning the actual testing phase with ALS patients, we performed a pilot

study with healthy subjects to fine-tune the overall project testing setup. Specifically,

a small group of healthy subject was administered a subset of the eBrain designed

sessions to generally assess feasibility, user-friendliness, pleasantness and fatigue.

Emotional aspects related to the experimental setting, as well as its usability, have

been evaluated, too. In this paper we report the results of this pilot study.

2 Materials and Methods

2.1 Subjects

Eight healthy subjects (4 females and 4 males) constituted the participants, ranging in

age between 25 and 39 (M: 31.75, SD: 5.946). They were all volunteers with a

schooling degree ranging from 13 to 24 years of education (M: 19, SD: 4.276). All the

subjects were experienced in the use of PC (50% fair or good and 50% excellent),

some of them (50%) declared to have already used Brain Computer Interface or an

Eye-tracker system, and more than half (62.5%) had already participated into EEG

experiments. Exclusion criteria were related to the states of participants’ cardiac,

optical, mental, and psychological health. Participants were asked not to drink

caffeine or alcohol and not to smoke prior to the experimental test to avoid any effects

of these substances on the central and autonomic nervous system.

2.2 System Setting

Test architecture (Figure 1) was composed of an eye-tracking system, and a BCI

device, both controlled by a laptop PC (HP DV3-4101SL, Hewlett Packard, USA),

connected to an external monitor, meant for the stimuli presentation (Display PC).

The BCI device module consisted of a g.USBAmp biosignal amplifier (Guger

Technologies, Graz, Austria), connected to an active electrodes head cuff

(g.GammaCap, Guger Technologies). 16 EEG channels were used (FZ, C3, C4, CZ,

CPZ, P3, P4, PZ, PO3, PO4, POZ, PO7, PO8, O1, O2, OZ); ground was placed in

FPZ, and reference was located on the left ear lobe.

The eye-tracker was an Eyelink-1000 (SR Research Ltd., Mississauga, Ontario,

Canada), consisting of a high-speed infrared camera and the related illuminator,

positioned just below the Display Monitor.

Tests were administered through two different software: a customized version of

BCI2000 (http://www.bci2000.org/BCI2000/Home.html) [41]; [42] for the BCI

assessment and a suitable custom software, developed within the project for the eye-

tracking evaluation.

2.3 Experimental Procedures

Subjects who met the experimental criteria were enrolled and tested at the Applied

Technology for Neuro-Psychology Laboratory, located at the Istituto Auxologico

Italiano in Milan. All subjects were required to sign a release form. Since the

experiment was constituted by two separated parts, the design was balanced between

the subjects. Immediately after the former session, subjects were asked to fulfill the

required questionnaires and to rest for 15 minutes.

After a calibration, required by both instruments, the neuropsychologist started the

cognitive assessment through a Phonemic Fluency test and a Semantic Fluency test

(as described below), adapted to the experimental conditions (see figure 1).

At the end of Eye-tracking session, the subjects had to copy a sentence twice

(FIAT ALAN FORD MERCEDES BENZ): the first time using a virtual keyboard

(ordered letters in rows from A to Z see figure 2a), the second time using a scrambled

keyboard (mixed position of the letters, see figure 2b). The total time of the

experiment was about two hours.

Fig. 1. Experimental procedure timeline for both BCI and ET sessions.

2.4 Psychological and Usability Self-report Questionnaires

2.4.1 STAI form Y Questionnaire

The Italian version of the STAI form Y questionnaire was used to assess changes in

two different types of anxiety, namely, anxiety detected as the subject's current state

(STAI-Y1, i.e. state anxiety) and anxiety detected as a reasonably stable trait of the

personality of an individual (STAI-Y2, i.e., trait anxiety). A total of four Self-reported

STAI-Y1 were gathered before and after both BCI and Eye-tracking sessions. One

self-reported STAI-Y2 was gathered five minute before the experimental session [22].

2.4.2 Self Assessment Manikin (SAM)

The Self Assessment Manikin (SAM) is a non-verbal pictorial assessment technique

that directly measures the pleasure, arousal, and dominance associated with subjects'

affective reactions [23]; [24]; [25]. A total of four Self-reported SAMs were gathered

before and after both BCI and Eye-tracking sessions.

Fig. 2. The used ET Keyboards and the Saccades plotted over them: (a) Standard Virtual

Keyboard, (b) Scrambled Keyboard, (c) Saccades on Standard Virtual Keyboard, (d) Saccades

on Scrambled Keyboard.

2.5 Usability Inventory Post-test Questionnaire

Since there are no usability validated tests for Eye-tracking and BCI systems, we

developed a questionnaire with 19 items based on the literature available [26-29]. Our

purpose was to evaluate the instruments' general usability, and the following specific

variables: fatigue, screen readability, perceived usefulness, and so on.

2.5.1 Neuropsychological Tests

We used a classic spoken letter-based word generation procedures (Phonemic

Fluency), such as the Controlled Oral Word Association test [30-34] and then

repeated the procedure with the Category Fluency, also known as Semantic Fluency

[30]. These tests have been recognized as the most sensitive tools in detecting

cognitive impairments in ALS patients [5].

For both the Semantic and the Phonemic Fluency in the BCI session we used to

measure the time taken by the subjects to think the word starting with the given

letters: “A” and "H" respectively. A timer was started by the researcher right after the

letter was presented to the subject. Then, the timer was stopped when the subjects

indicated to be ready to effectively start to write the word with the BCI system.

Finally the procedure was repeated for the Semantic Fluency, with the categories

”furniture” and "means of transport."

Concerning the Eye-tracking system we used an adapted version of Fluency Index

[6] to adjust for eye motor component. Thus the subjects were required to write all the

words starting with a specific letter (“Q” and “Z”) in exactly one minute (generation

condition). Then the subjects were asked to copy exactly the same words while a

researcher measured the time taken (control condition). The same procedure was

repeated for the Semantic Fluency task, with the categories “type of shoes” and

“cooking ingredients.”

The difference between the specified time for the generation condition and the

time taken for control condition was then calculated and used to determine the

Fluency Index, which represented the average time taken to think about each word

[6].

3 Results

Data were analyzed with the aid of the statistical software SPSS, version 17

(Statistical Package for the Social Sciences–SPSS for Windows, Chicago, Illinois,

USA). Due to the small sample size, nonparametric tests were preferred, even if

several measures showed a normal distribution (also according to Kolmogorov-

Smirnov test).

In the following paragraphs the main results of this preliminary study are

presented.

3.1 Fluency Tests

In BCI session, Phonemic Fluency average time in seconds was 6.42±2.76 and

Semantic Fluency average time in seconds was 4.08±1.94. Regarding Eye-tracking

session, we used the Fluency Index, as above described. Phonemic Fluency Index was

4.28±5.84 and the Semantic Fluency Index was 3.37±2.58. These data will be used to

assess future patients’ performances.

3.2 Behavioral Measures

Wilcoxon Signed Ranks Tests indicated no statistical differences for both the pre-post

STAI-Y1 and pre-post SAM scales, indicating that no negative affective state or

anxiety have been generated by the performance with BCI or ET. However a small

increase in anxiety was detected after the use of BCI.

3.3 Usability

As can be seen in Table 1, subjects recognize both systems as enough usable, but Eye-

tracker is perceived as more usable than BCI (statistical significance is calculated

with Wilcoxon Signed Ranks Tests).

Table 1. Average values of 7-point Likert scale items of the usability questionnaire.

ID Item Mean BCI Mean ET p

1 It is easy to use the device 5.25 6.25 .131

2 The instructions are clear 6.25 6.50 .157

3 Sometimes I wondered if I was selecting the right letter 2.25 1.75 .336

4 Letters on the screen are clear and sharp 5.25 6.88 .066

5 I felt in command of this device when I was using it 5.13 5.38 .916

6 I properly understood the instructions 6.63 6.63 1.000

7 Using this device was frustrating 2.38 1.88 .357

8 I felt tense at times when using this device 2.63 2.50 .931

9 The device requires too many steps to work 2.00 1.38 .197

10 New users will find this device easy to use 4.63 6.25 .038

11 It is easy to make the device do what I want 5.13 5.38 .666

12 The device did what I expected 5.25 5.63 .732

13 The device appears to be limited 2.88 2.38 .480

14 The equipment is comfortable 3.75 4.63 .167

15 Using the device was demanding and tiring 4.13 2.63 .071

16 The icon to correct answers was helpful (ET) - 5.88 -

17 An initial tutorial on the usage of the device would be helpful 3.25 2.75 .194

18 Using the device was amusing and exciting 4.25 5.25 .071

I believe that the device is fit to communicate in the presence

of disorders that prevent from communicating with the voice

5.13 6.75 .102

3.4 Anxiety Traits and Consequences

No correlations were found between STAI-Y2 and all the Fluency measures, for both

the BCI and the Eye-tracking.

On the other hand we found interesting correlation between STAI-Y2 and some

items of the usability questionnaire. In particular, the item 15 for the BCI (the

instruments has been demanding and tiring) is positive correlated with STAI-Y2 (r =

0.747, p = .033). In sum, usability is negatively correlated with the trait anxiety level.

3.5 Copying Text using Different Keyboards with ET

The average time to copy the same text with the standard virtual keyboard was

46.09±6.55. The average time for the same copy with the scrambled keyboard was

60.17±21.23. Wilcoxon Signed Ranks Tests indicated statistical significant

differences (Z = -2.201, p = .028).

4 Conclusions and Future Work

No studies have been performed so far to evaluate BCI and the eye-tracking system

for AAC and cognitive assessment in ALS. Our pilot study provided evidences for the

effectiveness and usability of these techniques. Specifically, the BCI/computerized

assessment could provide new insight into the understanding of cognitive deficits,

when applied to ALS patients, through the integration of multidisciplinary data:

neurophysiological, neuropsychological, behavioural and psychological.

The proposed study is characterized by at least two innovative aspects: (1) the

comparison between two promising technologies, one already extensively

investigated (ET), the other being a very promising candidate (P300 BCI), (2) the

adaptation of a computerized verbal fluency task for the neuropsychological

assessment of higher order cognitive functions in ALS patients.

Results showed a good usability of both instruments, better for Eye-tracking, but

promising for BCI too. Furthermore the strong negative correlation between trait

anxiety and perceived usability clearly shows that the higher the subject is anxious,

the more the instruments will be perceived as demanding, tiring, and difficult to use.

Finally, no affective effects on cognitive performances were revealed by the

psychological measures administered.

As expected, BCI calibration was a critical issue. The average score obtained after

the calibration process was 89.73%, leading to an accuracy of BCI system during the

tests of 81.72% (i.e. 18.28% of errors). However, the 6 subjects that obtained 100% of

correct calibration did very few errors in the testing phase. These data suggest that it

is crucial to extend the calibration phase in order to reach very high correct ratio

(close to 100%).

More, the choice of virtual keyboard used in the task, clearly influences the

performances obtained, showing a high potential use of such instruments for the

development of novel cognitive tests.

Finally, these preliminary results may have interesting implications for both

clinical practice (the availability of an effective tool for neuropsychological

evaluation of ALS patients) and ethical issues, the last one arising from a proper

assessment of cognitive ability preservation, in particular regarding relevant decisions

about medical treatments, economical and end-life issues.

Acknowledgements

This study has been made possible partially due to funds from the Lombardy Region

project "eBrain: BCI-ET for ALS (eBrain: BCI-ET nella SLA)."

References

1. Bak T. H., Hodges J. R., (2001). Motor neurone disease, dementia and aphasia:

coincidence, co-occurrence or continuum? J Neurol, 248(4):260-70.

2. Strong M. J., Lomen-Hoerth C., Caselli R. J. et al., (2003). Cognitive impairment,

frontotemporal dementia, and the motor neuron diseases. Ann Neurol, 54:S20-23.

3. Strong M. J., Grace G. M., Freedman M., Lomen-Hoerth C., Woolley S., Goldstein L. H.,

Murphy J., Shoesmith C., Rosenfeld J., Leigh P. N., Bruijn L., Ince P., Figlewicz D.,

(2009). Consensus criteria for the diagnosis of frontotemporal cognitive and behavioural

syndromes in amyotrophic lateral sclerosis. Amyotroph Lateral Scler, 10(3):131-46.

4. Zago S., Poletti B., Morelli C., Doretti A., Silani V., (2011). Amyotrophic lateral sclerosis

and frontotemporal dementia (ALS-FTD). Archives Italiennes de Biologie, 149: 39-56.

5. Phukan J., Pender N. P., Hardiman O., (2007). Cognitive impairment in amyotrophic lateral

sclerosis. Lancet Neurol, 6(11):994-1003.

6. Abrahams S., Leigh P. N., Harvey A., Vythelingum G. N., Grisé D., Goldstein L. H.,

(2000). Verbal fluency and executive dysfunction in amyotrophic lateral sclerosis (ALS).

Neuropsychologia, 38(6):734-47.

7. Kübler A., Birbaumer N., (2008). Brain-Computer interfaces and communication in

paralysis: extinction of goal directed thinking in completely paralysed patients? Clinical

Neurophysiology, 119:2658-2666.

8. Fabiani M., Gratton G., Karis D., Donchin E., (1987). Definition, identification and

reliability of the P300 component of the event-related brain potential. Ackles, P.K.,

Jennings, J. R., Coles, M. G. H. (Eds.). Advances in psychophysiology, 2. New York, NY:

JAI Press, 1-78.

9. Sellers E. W., Donchin E., (2006). A P300-based brain–computer interface: Initial tests by

ALS patients. Clin Neurophysiol,117:538-548.

10. Sellers E. W., Kübler A., Donchin E., (2006). Brain-computer interface research at the

University of South Florida Cognitive Psychophysiology Laboratory: the P300. Speller.

IEEE Trans Neural Syst Rehabil Eng, 14(2):221-4.

11. Dickhaus T., Sannelli C., Muller K. R., Curio G., Blankertz B., (2009). Predicting BCI

performance to study BCI illiteracy. BMC Neuroscience, 10.

12. Nijboer F., Sellers E. W., Mellinger J., Jordan M. A., Matuz T., Furdea A., Halder S.,

Mochty U., Krusienski D. J., Vaughan T. M., Wolpaw J. R., Birbaumer N., Kübler A.,

(2008). A P300-based brain-computer interface for people with amyotrophic lateral

sclerosis. Clin Neurophysiol, 119(8):1909-16.

13. Paulus K. S., Magnano I., Piras M. R. et al., (2002). Visual and auditory event-related

potentials in sporadic amyotrophic lateral sclerosis. Clin Neurophysiol, 113:853–61.

14. Hanagasi H. A., Gurvit I. H., Ermutlu N. et al., (2002). Cognitive impairment in

amyotrophic lateral sclerosis: evidence from neuropsychological investigation and event-

related potentials. Cognitive Brain Research, 14:234-244.

15. Gil R., Neau J. P., Dary-Auriol M. et al., (1995). Event-related auditory evoked potentials

and amyotrophic lateral sclerosis. Arch Neurol, 52: 890-896.

16. Ogawa T., Tanaka H., Hirata K., (2009). Cognitive deficits in amyotrophic lateral sclerosis

evaluated by event-related potentials. Clinical Neurophysiology, 120:659-664.

17. Schreiber H., Gaigalat T., Wiedemuth-Catrinescu U., Graf M., Uttner I., Muche R.,

Ludolph A. C., (2005). Cognitive function in bulbar- and spinal-onset amyotrophic lateral

sclerosis. A longitudinal study in 52 patients. J Neurol, 252(7):772-81.

18. Kübler A., Kotchoubey B., Kaiser J. et al., (2001). Brain-Computer Communication:

unlocking the locked in. Psychological Bullettin, 127(3): 358-375.

19. Cipresso P., Meriggi P., Carelli L., Solca F., Meazzi D., Poletti B., Lulé D., Ludolph A. C.,

Riva G., Silani V., (2011), The combined use of Brain Computer Interface and Eye-

Tracking technology for cognitive assessment in Amyotrophic Lateral Sclerosis, 5th

International Conference on Pervasive Computing Technologies for Healthcare

(PervasiveHealth) (2011): p. 320-324.

20. Wolpaw J. R., Schalk G., McFarland D. J., Hinterberger T., Perelmouter J., Godde B.,

Birbaumer N., Pfurtscheller G., BCI2000: a general purpose brain–computer interface

system. Soc Neurosci Abstr 2000c;26:1229.

21. Schalk G., McFarland D. J., Hinterberger T., Birbaumer N., Wolpaw J. R., BCI2000:

development of a general purpose brain–computer interface (BCI) system. Soc Neurosci

Abstr 2001;27:168.

22. Spielberger C. D., Goursuch R. I. and Lushene R. E., (1970) Manual for state and trait

anxiety inventory (self-evaluation questionnaire). Consulting Psychologist Press, Palo Alto,

CA.

23. Lang, P. J., (1995). The emotion probe. Studies of motivation and attention. The American

Psychologist, 50(5), 372-385.

24. Lang P. J., Behavioral treatment and bio-behavioral assessment. In: Sidowski J. B., Johnson

J. H., Williams T. A., editors. Technology in mental health care delivery systems,

Norwood, NJ: Ablex, 1980, pp. 119–37.

25. Bradley, M. M. and Lang, P. J., (1994). Measuring emotion: The self-assessment manikin

and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry,

25, 49-59.

26. Nielsen, J., (1992). The usability engineering life cycle. IEEE Computer 25,3 (March), 12–

22.

27. Nielsen, J., (1993). Usability Engineering, Academic Press, Boston, MA.

28. Velichkovsky, B. and Hansen, J. P., (1996). New technological windows into mind: There

is more in eyes and brains for human-computer interaction. In Proc. CHI 1996, 496-503.

29. Kitamura, Y., Yamaguchi, Y., Imamizu, H., Kishino, F. and Kawato, M., (2003). Things

happening in the brain while humans learn to use new tools. In Proc. CHI 2003, 417-424.

30. Abe K., Fujimura H., Toyooka K., Sakoda S., Yorifugi S., Yanagihara T., Cognitive

function in amyotrophic lateral sclerosis. Journal of Neurological Sciences 1997;148:95-

100.