Physiologically Attentive User Interface for Robot Teleoperation

Real Time Emotional State Estimation and Interface Modification using

Physiology, Facial Expressions and Eye Movements

Gaganpreet Singh

, Sergi Bermúdez i Badia

1,2

, Rodrigo Ventura

and José Luís Silva

1,4

Madeira-ITI, Universidade da Madeira, Funchal, Portugal

Faculdade de Ciências Exatas e da Engenharia, Universidade da Madeira, Funchal, Portugal

Institute for Systems and Robotics, Instituto Superior Técnico, Universidade de Lisboa, Lisboa, Portugal

Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL, Lisboa, Portugal

Keywords: Psychophysiology, Biosignals, Bitalino, Robot Teleoperation, Facial Emotions, Electroencephalography,

Electrocardiogram, Electrodermal Activity, Electromyography, Eye Tracking, Sikuli, Picture-driven

Computing, Automation, ISI, Emotions.

Abstract: We developed a framework for Physiologically Attentive User Interfaces, to reduce the interaction gap

between humans and machines in life critical robot teleoperations. Our system utilizes emotional state

awareness capabilities of psychophysiology and classifies three emotional states (Resting, Stress, and

Workload) by analysing physiological data along with facial expression and eye movement analysis. This

emotional state estimation is then used to create a dynamic interface that updates in real time with respect to

user’s emotional state. The results of a preliminary evaluation of the developed emotional state classifier for

robot teleoperation are presented, along with its future possibilities are discussed.

1 INTRODUCTION

Due to many fold increase in computing capabilities,

we have seen tremendous evolution in Human-

Computer Interaction (HCI). And through

technological improvements and robotics evolution;

we are witnessing another form of interaction which

is between Humans and Robots; and widely known

as Human-Robot Interaction (HRI). No matter if it is

HCI or HRI, the ways we are interacting with

machines have evolved to such an extent that

science is now looking for methods that help

understand human intentions without much need of

physical input from humans.

From the emergence of computers to the

development of personal computers, and then

becoming an ubiquitous entity, the relationship

between humans and computers shifted from many-

to-one to one-to-one, and now it is one-to-many.

This technological shift brings in the demand for

smarter Human-Computer Interfaces.

To optimize HCI, Vertegaal (2003) proposed a

framework for Attentive User Interfaces (AUI). AUI

uses sensing, communication, augmentation, control,

and availability of human to strategically optimize

communication between Humans and Machines.

AUIs designed by different authors (Vertegaal,

2003; Siewiorek et al., 2003; Vertegaal et al., 2006)

use sociable forms of interaction by sensing user’s

attention levels for their surroundings and more

preciously for the Interface itself.

However, current AUIs depend on overt

measurements of user's attention, such as eye

contact, which may not always accurately indicate

user’s availability for notifications or interruptions.

Although overt measures of user’s attention may tell

us that a user is performing a given task, they do not

necessarily indicate the covert state of mind.

Due to this one-to-many relationship between

humans and computer systems, traditionally

designed approaches are not capable enough to

convey information from these devices to humans in

a precisely uninterrupted way. On the opposite, these

information hungry devices trigger un-timely

notifications and information delivery, and they are

becoming heavier and more demanding with time

(Dirican & Gokturk, 2009).

294

Singh, G., Badia, S., Ventura, R. and Silva, J.

Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modiﬁcation using Physiology, Facial Expressions and Eye Movements.

DOI: 10.5220/0006733002940302

In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 4: BIOSIGNALS, pages 294-302

ISBN: 978-989-758-279-0

Alongside, robot teleoperation also caught into

this attention’s demand created by interface and

robot operations. This eventually creates lots of

workload and stress on operators, and sometimes

operators also experience boredom, interest loss, and

focus issues. Drones like Global Hawks from US Air

Force have such a sophisticated system that they

need more attention and mental presence than

normally flying a plane. In other situations these

drones do not require attention and mental focus

every single time, which creates windows of

unawareness and lack of attention which decreases

performance and could cause problems. Secondly,

the complexity of these systems could be very high

in some situations, leading to very high mental

workload and induced stress on operators.

Fortunately, we have improved cognitive

abilities to understand covert emotional states that

are particularly not possible for current Graphic User

Interfaces (GUIs) or Attentive User Interfaces

(AUIs) (Dirican & Göktürk, 2011).

Psychophysiological activities provide a quiet,

hidden, and implicit way to understand cognitive

and affective states of users with respect to their

mind-body relationship (Dirican & Göktürk, 2011).

Human physiology is highly affected by the activity

of the Central Nervous System (CNS) and the

Autonomic Nervous System (ANS), and reflects

physiology in the form of physical signals generated

by human body in real time (Sapa, 2011; Dirican &

Göktürk, 2011), which could be helpful in telling

emotional state of a person.

We explored the field of Psychophysiology to

understand covert states of human mind alongside

integrated this with overt measurements of facial

expressions and eye movements, and prepared an

Artificially Intelligent system to precisely detect

three Emotional states (Resting, Stress, and

Workload).

These emotional predictions were then used in

real time to create a Physiologically Attentive User

Interface (PAUI) that changes dynamically with

respect to the emotional state of the person in real

time.

However, this generated PAUI developed over

an older GUI to reduce the complexity (e.g. reducing

the amount of information provided) and to increase

both usability and development flexibility for a

closed system. In which the older GUI was a frozen

12 years old interface with very complex user

interaction and feedback view. This new interface

reduces the complexity of the older one by

displaying only the more relevant information

(reducing unnecessary user’s cognitive overload)

and updates itself in real time with respect to the

emotional state of the person. In addition, it will also

communicate with the older GUI, via a picture-

driven computing approach e.g. (Silva et al., 2016),

to eliminate the need of creating a new interactive

system from scratch.

Following in this paper we have discussed

current state of physiologically driven interfaces and

Human-Machine interactions. Then explained the

solution we developed for PAUI creation and the

framework designed. Then next, emotion

classification processes, experiments, and there

results are discussed. It is then followed by the

conclusion and its future perspective.

2 REVIEW

Due to emotional state awareness capabilities of

Psychophysiological measures, they are catching lot

of attention these days in areas like Autonomous

systems, Military, Medicine, among others. Few of

them relevant to the project were studied and their

findings are discussed below.

Bulling (2016) provides an analytical and

projective view on current and future aspects of User

Interfaces, with an insight to the possibilities and

requirements for Pervasive Attentive User

Interfaces. User interfaces will shift their focus from

being an attention demanding to attention managing

systems; interfaces adapt for amount, type, and time

of information delivery on the basis of current

attention capacities of the users. Bulling (2016)

defined Unobtrusiveness, Accuracy, Large scale,

Long-lividness, Seamlessness, and Context

awareness are 6 important categories that defines

new Pervasive Attentive User Interfaces.

Chen & Vertegaal (2004) used LF spectral

components for mental workload and analyzed EEG

for motor activity to find four distinguish states of

user, and use them to predict the availability of the

user for interrupts. These four states have

interruption costs for speech and motor related

activities. And by using user’s physiological state

and cost of interruption (calculated by user’s

preferences for mode of interrupt for email, IM, and

calls in all four states), system decides if the user has

to be interrupted or not. First state of this system

exhibits very lower degree of attention, in which

user is not actively engaged with any task and could

be interrupted for having relatively very low

interruption cost. However, this was not generalized

with other relaxing states where interruption cost

could be high. Second state has low interruption cost

Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modiﬁcation using

Physiology, Facial Expressions and Eye Movements

295

for audio related interrupts but has high interruption

cost for motor related interrupts like messaging. This

state is associated with transit activities like walking

or running. Third state is mental engagement while

at rest which results into high cost for auditory

interrupts that could interface with mental state of

the user. And the fourth state is of higher activity

engagement in which interrupt cost of any kind is

high and should not be disturbed.

A Human-Computer Interface (HCI) was

developed by (Chapin et al., 1999; Wessberg, 2000)

in Duke University to establish communication

between a Monkey’s brain and a Robot arm. To

achieve this communication they used multiple EEG

electrodes implanted over a greater area of monkey’s

brain. Neural activity of large population of

monkey’s brain was recorded and then decoded the

arm movements out of them. This information was

then used to reproduce the movements in robot arm.

Another example of Brain-Machine

communication was demonstrated jointly by Honda

Research Institute Japan, Advanced

Telecommunications Research Institute International

(ATR) and Shimadzu Corporation in March 2009, in

which a Robot was controlled only by Human

thoughts. They measured electric signals and blood

flow changes in the brain while imagining body part

movements and used these to predict user’s thought

process. These predicted motions are then supplied

to Honda’s ASIMO humanoid robot to perform

similar movements like raising its arm. More than

90% of accuracy rate was achieved (Zhang et al.,

2010).

Caproni et al. (2009) has developed a

comprehensive hemodynamic pattern classification

framework to enhance Human-Robot Interaction

(HCI) for medical robotics using Near-Infrared BCI.

Caproni et al. (2009) studied different simulations

for Motor Imagery (MI) and Non-Motor Imagery

(NMI) frameworks. Simulation combinations

depends on three channel combinations i.e. left,

right, and all channels; two classifier i.e. Support

Vector Machine (SVM) and AdaBoost; and three

aggregation policies i.e. Majority Voting, Weighted

Majority Voting, and Correcting Classifiers. Out of

which they found NMI as a best performer. After

scrutinizing all of their experiments and their results,

Caproni et al. (2009) concluded Near InfraRed

Spectroscopy (NIRS) based Brain Computer

Interfaces has a huge potential to help enhance

existing Human-Machine Interfaces.

3 PAUI

3.1 Approach

We created a basic Physiologically Attentive User

Interface (PAUI) to read and understand user’s

Psychophysiology in real time with an intention to

classify three different emotional states (Resting,

Stress, and Workload) of a person while

teleoperating a robot. These classification results are

then use to change the interface in such a way that

improves user performance in the task and ease the

process of robot teleoperation.

3.2 Apparatus Used and Placement

We are using Bitalino by Plux (Bitalino, 2017) for

reading biosignals that are Electroencephalography,

Electrocardiogram, Electrodermal activity, and

Electromyography. For eye tracking, we use Tobii

4c from Tobii Technologies (Tobii Technologies,

2017) and a normal webcam to extract facial

emotions.

Once the person is at the station s(he) will be

attached with Ag/AgCl electrodes under right

clavicula (Plus), under left musculus pectoralis

major (Minus), and under left clavicula (Neutral) for

ECG’s best suggested placement by (Němcová et al.,

2016); for EDA two electrodes were used on left

palm; for EMG negative and positive electrode are

placed at Abductor pollicis brevis muscle of left

hand and reference electrode at left arm’s Head of

ulna; and for EEG negative and positive electrodes

were placed at forehead and reference electrode at

left earlobe.

3.3 Architecture

PAUI application’s architecture shown in figure 1 is

divided into three sub modules that work alongside

to achieve overall goal of creating Physiologically

Attentive User Interface (PAUI) for robot

teleoperation. The three sub modules are Emotional

State Estimator (ESE), Attentive User Interface

(AUI), and System Integrator (SI).

ESE interacts with external hardware modules to

extract covert and overt data of the user and process

that for emotion prediction. This predicted emotional

state is then fed to AUI that makes changes to its

interface with respect to that. And SI is helping in

filling the communication gap between old GUI and

new PAUI. Moreover, communication between

Hardware layer – ESE and ESE – AUI is one way,

but between AUI – SI and SI – Old GUI is two way.

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

296

3.3.1 Emotional State Estimator (ESE)

This module is sub divided into 4 parallel

threads: Bitalino thread extracts data at 1000 Hz for

processing physiological signal; Camera thread

processes camera images and extracts facial

emotions at 15 Hz; Tobii thread extracts data at 90

Hz for tracking eye movements; and the Classifier

thread runs at 2000 Hz that reads data from Bitalino,

Camera, and Tobii thread, and performs emotion

extraction and provides predicted emotion.

Bitalino thread processes ECG signal for Heart

Rate (HR), Heart Rate Variability i.e. Standard

Deviation of Normal to Normal (SDNN) and Root

Mean Square of the Successive Differences

(RMSSD), and Frequency components i.e. Very

Low Frequency (VLF from 0.0033 to 0.04), Low

Frequency (LF from 0.04 to 0.15 Hz), and High

Frequency (HF from 0.15 to 0.4 Hz). It processes

EEG for Delta (0.5 – 3.5 Hz), Theta (3.5 – 8 Hz),

Alpha (8 – 13 Hz), Beta (13 – 30 Hz), Gamma (30 –

45 Hz), and Engagement (Engagement = Beta /

(Alpha + Theta)) suggested by McMahan et al.

(2015). Processes EDA for Skin Conductance Level,

Skin Conductance Response. And EMG is processed

for Muscle Fiber Excitation (MFE). Table 1 contains

the list of parameters extracted from each device.

The camera thread uses common webcam and

Emotion SDK from Affectiva (Affectiva, 2017) to

processes image frames and extracts emotions from

Figure 1: PAUI Architecture.

Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modiﬁcation using

Physiology, Facial Expressions and Eye Movements

297

faces present in the frames. In that we are extracting

9 facial emotions, 21 facial expressions, and facial

orientation information.

Tobii thread is continuously monitoring eye

movements on the screen and keeps on updating

custom designed data set for fixations map on the

screen. It contains comprehensive information of

fixation map like average fixation, biggest fixation

at, among fixation specific information like number

of fixations at particular location, fixation coming

from and fixation going to, and so on.

Classifier thread works above all three threads, it

takes data from them approximately every 500 micro

seconds, do the average of data of 3 seconds and use

this averaged data with trained Support Vector

Machine (SVM) classifier for Emotion prediction

out of three trained emotional states (i.e. Resting,

Stress, and Workload).

Table 1: Extracted parameters from all three sensors.

Sensor

Category

Features

Bitalino

ECG

▪ HR

▪ SDNN

▪ RMSSD

▪ VLF

▪ LF

▪ HF

EEG

▪ Delta

▪ Theta

▪ Alpha

▪ Beta

▪ Gamma

▪ Engagement

EMG

▪ Number of Peaks

▪ Total Peak Time

▪ Max Peak Magnitude

▪ Current Peak Magnitude

EDA

▪ SCL

▪ SCR

Tobii

General Fixation

Information

▪ Number of Fixations

▪ Total Time

▪ Total Fixation Duration

▪ Average Fixation

Duration

▪ Repeated Fixations

▪ Biggest Fixation At

▪ Maximum Visited Counts

▪ Maximum Visited At

Fixation Map

*Containing

information for

each fixation

▪ Number of Visits

▪ Start Time

▪ Fixation Duration

▪ Total Fixation Duration

▪ Total Interval Between

Visits

▪ List of Locations Coming

From and its count

▪ List of Locations Going

To and its count

Camera

Emotions

▪ Joy

▪ Fear

▪ Disgust

▪ Sadness

▪ Anger

▪ Surprise

▪ Contempt

▪ Valence

▪ Engagement

Expressions

▪ Smile

▪ Inner Brow Raise

▪ Brow Raise

▪ Brow Furrow

▪ Nose Wrinkle

▪ Upper Lip Raise

▪ Lip Corner Depressor

▪ Chin Raise

▪ Lip Pucker

▪ Lip Press

▪ Lip Suck

▪ Mouth Open

▪ Cheek Raise

▪ Dimplier

▪ Eye Widen

▪ Jaw Drop

▪ Lip Tighten

▪ Lip Stretch

▪ Smirk

▪ Eye Closure

▪ Attention

Face Orientation

▪ Pan

▪ Tilt

▪ Yaw

3.3.2 Attentive User Interface (AUI)

This is the interactive interface with which user is

meant to interact and it changes with respect to users

psychophysiological state predicted by Emotional

State Estimator (ESE). It keeps on reading

psychophysiological state predicted by ESE along

with data provided by Tobii, and performs required

changes in its design along with sending required

operations to SI for old GUI.

3.3.3 System Integrator (SI)

System Integrator (SI) is the communication bridge

between new Physiologically Attentive User

Interface (PAUI) and any old GUI (used for robot

teleoperation in our example, figure 2). It needs to

perform two basic functionalities in between PAUI

and old GUI. One is to extract data from old GUI in

a reliable and continuous manner to provide working

information to the user (Feedback Extractor sub-

layer). And secondly, it needs to take action

commands from AUI and perform required activities

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

298

on old GUI using Sikuli’s task automation properties

(Sikuli sub-layer).

Figure 2: Old GUI for robot teleoperation.

3.4 Experiment Setup

3.4.1 Virtual Environment Experiments

Experiments were divided into two categories:

Virtual Environment and Robot Teleoperation. To

get initial understanding of the data and to perform

preliminary tests, we created a virtual setup using

games that help generating emotional stimulus in

subjects.

In virtual environment experiments the subject

were given 2 minutes of relaxing time in the

beginning, afterwards s(he) needs to perform

Relaxing task for 5 minutes. It was then followed by

self-assessment in NASA-TLX and SAM.

Afterwards, the person was either put on Workload

or Stress task randomly.

In the Workload task, subjects were asked to

perform 3 tasks in Rigs of Rods (Rigsofrods.org,

2017) for 5 minutes each, in which difficulty was

increased linearly. These 3 sessions of workload

were separated by 0.30 minute of break and self-

assessment on NASA-TLX.

The Stress test is also divided into 3 sessions on

a modified Tetris game to make it very hard for

whole time play. All three sessions of Stress were

separated by 0.30 minutes of self-assessment on

SAM and break alongside.

After finishing first session of Workload or

Stress task, subjects were introduced to Relaxing

session for 5 minutes and then again put on either

Workload or Stress task. The Workload and Stress

sessions were pseudo randomized in such a way that

if the first session is of Workload then the second

should be of Stress and same should be other way

around.

3.4.2 Robot Teleoperation Experiments

After conducting preliminary experiments on virtual

environments, we performed experiments on robot

teleoperation while imitating search and rescue

operations of Fire fighters.

In which Resting was performed by driving the

robot from one end to other end in a long room for

five minutes at minimum speed, to simulate

inactivity and lack of mental and physical demand.

Then in the Stress task subjects need to teleoperate

the robot through a very difficult environment and

have to finish this task within 5 minutes. And in the

Workload task, subjects have to search for five items

in the environment alongside answering basic

arithmetic operations.

However, to keep things unbiased we

randomized the whole testing procedure. Each

subject has to perform 2 sessions of each task in a

randomly controlled way. A home like test setup

was used for the experiments that contains a

bedroom, living room, and a Kitchen; installed in

our lab. Alongside, this home like test setup, we also

used some parts of the lab for these experiments.

And for Stress tests, 2 specially designed areas in the

lab were used to intensify task difficulty and to

elevate stress.

Figure 3: Graph containing engagement data from resting test in virtual environment experiments, in which engagement

value shows very less intensity.

Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modiﬁcation using

Physiology, Facial Expressions and Eye Movements

299

Figure 4: Graph containing engagement data from stress test in virtual environment experiments, in which engagement

value shows very high intensity.

After attaching all required electrodes and test

them thoroughly, each subject then put onto

realisation session, in which they introduced with

the tests and robot controls, and let them play with

the robot for five minutes. Then out of 6 sequential

combinations of resting, stress, and workload test, a

random combination was chosen and performed

with five minutes of wash away time in between

each session, in which they also needed to fill

NASA-TLX questionnaire for workload and 5 scale

Self-Assessment Manikin (SAM) containing

Valence and Arousal for stress. After completing the

combination of three tests, another combination was

selected out of remaining 5 combinations. Then for

the next subject only remaining combinations were

used and this continued until all 6 combinations

were used.

As proof of concept for this project, five subjects

participated and performed two sessions of each task

(Resting, Stress, and Workload) by each subject.

4 CLASSIFICATION AND

RESULTS

For benchmarking of our system we performed K-

Nearest Neighbour (KNN) and Support Vector

Machine (SVM) classification on data from both

Virtual Environment Experiments (VEE) and Robot

Teleoperation Experiments (RTE). And out of all

extracted features we used 46 features (without

normalization or scaling) for each vector space

As the physiological data is dependent on task

and varies from person to person, we try to evaluate

both the conditions. The data from VEE and RTE

were arranged into three different categories: Task

Specific (Gaming and Teleoperation) and Person

Specific, which then classified and evaluated.

In Task Specific, data from VEE and RTE were

trained and tested separately with SVM and KNN. In

which, 70% of VEE data was used for training SVM

and KNN and remaining 30% of VEE data was used

for testing, similar training and testing percentage

was used with RTE data. Thus the classification

results of SVM outperformed KNN in both the tasks.

With VEE data SVM gave 80.00% of accuracy and

KNN gave 77.63% of accuracy. And with RTE data

SVM gave 84.75% of accuracy and KNN gave

79.84% of accuracy.

In Person Specific, data of single subject from

RTE was used. In which, both SVM and KNN were

trained with 70% data and tested with remaining

30% of data. As the data represent only one single

person, classifiers performed relatively better than

Task Specific, and gave 88.37% of accuracy by

SVM and 82.95% of accuracy by KNN. Please refer

table 2 for classification results.

Table 2: Classification Results.

DATA

SVM

KNN

Task Specific

(VEE)

80.00%

77.63%

Task Specific

(RTE)

84.75%

79.84%

Person Specific

88.37%

82.95%

As we conducted a validatory research, to

support the framework for Physiologically Attentive

User Interface, small amount of training data was

used which may have interfered with the

classification accuracy. Nevertheless, SVM gave

expectedly good results to support the framework for

real time emotional state processing.

Alongside, the engagement data from resting and

stress tests conducted in virtual environment

experiments shows clear differences in the patters

and intensity in both the measurements. Intensity of

engagement value in stress test is relatively very

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

300

high as compared to resting, and it also stayed high

in whole stress session, shown in figure 3 and 4.

5 CONCLUSIONS AND FUTURE

PERSPECTIVE

User’s psychophysiological state was measured and

predicted in real time and autonomy is provided to

the system to improve its interface dynamically with

respect to the mental workload and stress level on

the user. A PAUI was created, that performs

dynamic updations to its interface and helps in

decelerating the effects of workload and stress.

Moreover, the classification findings are quite

impressive. We have explored different aspects of

psychophysiology and combined them with external

emotional and attentional clues. Getting 88.37% of

accuracy in Person specific data and 84.75%

accuracy in Task specific data with this small

amount of training samples gives a valid indication

of having huge potential of improvement.

Current findings clearly suggest that the use of

Deep learning techniques could be a promising

measure to achieve higher degree of accuracy in

emotion classification.

Future aspects of this research are with the

improvements in emotion classification techniques

with current state of the art classifiers. One

important field to scrutinize is with Recurrent Neural

Networks that could be helpful in understanding the

changing patterns of the data and make prediction on

them. And to introduce more emotional states for

classification which helps in bring more dynamicity

and understandability to PAUI.

ACKNOWLEDGEMENTS

This work was supported from Fundação para a

Ciência e a Tecnologia (FCT, Portugal), through

project UID/EEA/50009/2013.

REFERENCES

Affectiva. 2017. SDK & API. [online] Available at:

https://www.affectiva.com/product/emotion-sdk/

[Accessed 7 Nov. 2017].

Bitalino, 2017. EEG Sensor | Biomedical Sensors |

BITalino Board Kit. [online] Bitalino.com. Available

at: http://bitalino.com/en/plugged-kit-bt [Accessed 7

Nov. 2017].

Bulling, A., 2016. Pervasive Attentive User

Interfaces. IEEE Computer, 49(1), pp.94-98.

Caproni, M., Orihuela-Espina, F., James, D.R., Menciassi,

A., Dario, P., Darzi, A.W. and Yang, G.Z., 2009,

October. An analysis framework for near infrared

spectroscopy based brain-computer interface and

prospective application to robotic surgery.

In Intelligent Robots and Systems, 2009. IROS 2009.

IEEE/RSJ International Conference on(pp. 2143-

2148). IEEE.

Chapin, J.K., Moxon, K.A., Markowitz, R.S. and

Nicolelis, M.A., 1999. Real-time control of a robot

arm using simultaneously recorded neurons in the

motor cortex. Nature neuroscience, 2(7), pp.664-670.

Chen, D., 2006. The Physiologically Attentive User

Interface Towards A Physiological Model of

Interruptability (pp. 2836-2836). Queen's University.

Chen, D. and Vertegaal, R., 2004, April. Using mental

load for managing interruptions in physiologically

attentive user interfaces. In CHI'04 extended abstracts

on Human factors in computing systems (pp. 1513-

1516). ACM.

Dirican, A.C. and Gokturk, M., 2009. Attentive

computing. In Human-Computer Interaction. InTech.

Dirican, A.C. and Göktürk, M., 2011. Psychophysiological

measures of human cognitive states applied in human

computer interaction. Procedia Computer Science, 3,

pp.1361-1367.

McMahan, T., Parberry, I. and Parsons, T.D., 2015.

Evaluating player task engagement and arousal using

electroencephalography. Procedia Manufacturing, 3,

pp.2303-2310.

Němcová, A., Maršánová, L. and Smíšek, R., 2016

RECOMMENDATIONS FOR ECG ACQUISITION

USING BITALINO.

Rigsofrods.org. 2017. Rigs of Rods Soft Body Physics

Simulator. [online] Available at: https://rigsofrods.org/

[Accessed 7 Nov. 2017].

Sapa, M., 2011. Psychological state estimation from

physiological recordings during robot-assisted gait

rehabilitation. Journal of rehabilitation research and

development, 48(4), p.367.

Siewiorek, D.P., Smailagic, A., Furukawa, J., Krause, A.,

Moraveji, N., Reiger, K., Shaffer, J. and Wong, F.L.,

2003, October. SenSay: A Context-Aware Mobile

Phone. In ISWC (Vol. 3, p. 248).

Silva, J.L., Ornelas, J.D. and Silva, J.C., 2016, June. Make

it ISI: interactive systems integration tool.

In Proceedings of the 8th ACM SIGCHI Symposium

on Engineering Interactive Computing Systems (pp.

245-250). ACM.

Tobii Technologies. 2017. Get your Tobii Eye Tracker 4C.

[online] Available at: https://tobiigaming.com/eye-

tracker-4c/ [Accessed 7 Nov. 2017].

Vertegaal, R., 2003. Attentive user

interfaces. Communications of the ACM, 46(3),

pp.30-33.

Vertegaal, R., Shell, J.S., Chen, D. and Mamuji, A., 2006.

Designing for augmented attention: Towards a

Physiologically Attentive User Interface for Robot Teleoperation - Real Time Emotional State Estimation and Interface Modiﬁcation using

Physiology, Facial Expressions and Eye Movements

301

framework for attentive user interfaces. Computers in

Human Behavior, 22(4), pp.771-789.

Wessberg, J., Stambaugh, C.R., Kralik, J.D., Beck, P.D.,

Laubach, M., Chapin, J.K., Kim, J., Biggs, S.J.,

Srinivasan, M.A. and Nicolelis, M.A., 2000. Real-time

prediction of hand trajectory by ensembles of cortical

neurons in primates. Nature, 408(6810), pp.361-365.

Zhang, B., Wang, J. and Fuhlbrigge, T., 2010, August. A

review of the commercial brain-computer interface

technology from perspective of industrial robotics.

In Automation and Logistics (ICAL), 2010 IEEE

International Conference on(pp. 379-384). IEEE.

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

302