Psychophysiological Measurements in Real Working Environments

Wireless EEG Study of the Operators’ Vigilance

Pavle Mijović, Ivan Gligorijević, Evanthia Giagloglou and Ivan Mačužićand Branislav Jeremić

Faculty of Engineering Sciences, University of Kragujevac, Sestre Janjić 6, Kragujevac, Serbia

1 STAGE OF THE RESEARCH

The vigilance decrement and inability of the industry

worker to sustain attention during a task can lead to

errors in operating, which could further lead to

dangerous situations including catastrophic events

with fatalities. Therefore, measuring operators’

vigilance level, while performing everyday

monotonous repetitive task, is of crucial interest.

The current project is related to innovation

through human factors in risk analysis and

management at the work place (InnHF project,

http://www.innhf.eu). The term innovation is

justified by hypothesis that it is possible to obtain

higher degree of operators’ safety and workers well-

being by on-line measurement of operators’

vigilance state, using lightweight and wireless sensor

systems. Extensive literature review in the fields of

neuroergonomics, human factors and ergonomics

(HFE), psychophysiology and biomedical signal

processing preceded the current research in order to

approach this problem in the most appropriate way.

At this point, we believe that it is possible to obtain

reliable measure of operators’ vigilance state, while

performing repetitive task, by continuous on-line

recording of the brain signals using compact and

lightweight wireless EEG system.

The initial experiments have been conducted,

using novel and state-of-the-art lightweight wireless

EEG system (SMARTING, made by mBrainTrain

LLC, Serbia), which confirmed the ability of the

device to obtain reliable, artefact-free recordings.

The signal strength, judged by visual inspection of

known eye-blink signatures and clearly visible alpha

activity, proved strong enough for the proposed

psychophysiological measurement of operators’

vigilance state. These "entry" experiments were

carried out using the open-source open ViBE

software package (http://openvibe.inria.fr).

Further, we have created the experimental set-up

of the improvised "lean" workplace, where the real-

life example from one of our industry partners is

authentically replicated. This serves to examine the

psychophysiological correlates of the operators’

vigilance state, while carrying out everyday

repetitive jobs.

We have also identified research paradigms and

the next step is to perform high-density EEG

measurements in these realistic work conditions,

identifying the physiological signatures of vigilance

state of the operator and offer tools for their real-

time detection.

2 OUTLINE OF OBJECTIVES

Neuroergonomics is the study of brain and behavior

at work (Parasuraman, 2003). It is a novel and

interdisciplinary area or research that merges the

disciplines of neuroscience and human factors and

ergonomics in order to maximize the benefits of

each (Parasuraman and Rizzo, 2007).

With the recent technological advancements it

became possible to move the EEG measurement

from the strictly controlled laboratory conditions

(movement-constrained behavior), to the real-life

environments where subjects are allowed to

naturally walk outdoor wearing wireless EEG device

(Debener et al., 2012). Therefore, with these new

wireless EEG systems it becomes possible to merge

EEG measurements with the guiding principle of

neuroergonomics, and examine how the brain carries

out the complex everyday work tasks, and not just

simple and artificial laboratory task (Parasuraman

and Rizzo, 2007).

The first objective of our work is to examine the

psychophysiological correlates of vigilance

decrement in the improvised, but highly realistic

workplace by using the wireless EEG system in or

der to avoid any constrains to physical actions of the

operators.

After the initial identification of these correlates,

the next step is to develop and implement real-time

vigilance estimation mechanisms.

Mijovi

c P., Gligorijevi

c I., Giagloglou E., Ma

cuži

cand I. and Jeremi

c B..

Psychophysiological Measurements in Real Working Environments - Wireless EEG Study of the Operators’ Vigilance .

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

Final stage assumes applying knowledge from

the improvised workplace repeating the experiments

in the real working environment featuring "lean"

production procedures. This is planned in

collaboration with the industrial partners from the

InnHF project (FIAT Automobiles Serbia and

Tetrapak Serbia). This brings twofold benefit: on the

side of research, it will be possible to obtain useful

information on the cognitive state of the operator

without disrupting the everyday work routine; on the

industry side, new guidelines and constrains for

increasing the workers vigilance levels are expected

to improve the ergonomic of the workplace and

consequently, the operators’ and equipment safety,

bringing reduction in costs and maximizing

production efficiency.

Another side objective is to upgrade the

vigilance monitoring system to create a brain

computer interface (BCI) that will allow the

operators’ safety action in cases that are critical from

the safety point of view. Also, this system is to

serve for control of robotic actions for the places

hardly reachable by the hand of an operator. It is

well known that in the industries with high

production output it is not always possible to design

the production line that is completely user-friendly,

due to robustness of the machinery. Therefore, the

BCI application for the safety actions would be a

highly desirable solution: the operator would use

comfortable, completely wireless EEG system that

would not require any physical action.

Finally, we believe that we will be able to

identify the optimal sensor positions using high-

density recordings brought in by our current wireless

EEG system. This is expected to enable reduction of

the number of necessarily used electrodes, which

could in turn offer guidelines for future industrial

vigilance and BCI sensor system that is lightweight,

comfortable, completely mobile and ready to be

used by workers. Therefore, we aim to contribute to

creation of the state-of-the-art on-line monitoring

and interaction system for the operators’ in the

industry. This way, the operators’ potential errors

due to decrement of alertness level could be

prevented, leading to decrease of the industrial

accidents and incidents that are in the most cases

caused by the operators’ lapses in sustained

attention.

The outline of the objectives of this work is

graphically represented in Figure 1.

Figure 1: Outline of the research objectives.

3 RESEARCH PROBLEM

3.1 Vigilance Monitoring

Regarding safety assessment, HFE is concerned with

the elimination, reduction, or mitigation of human

error. Human error is typically categorized as slips

(errors in actions), lapses (errors in memory), or

mistakes (errors in applying knowledge; Reason,

1990) and is often cited as a factor in up to 80% of

accidents (Wiegmann and Shappell, 2001).

The critical role that is assigned to HFE in design

and safety assessment depends on the widespread

use of automation and its impact on human errors

(Cacciabue, 2004). The automation is often

proposed as a method for removing or reducing

human errors in a system and it shifted the roles of

operators from active controllers to that of system

supervisors who serve in fail-safe capacity only

when problem arise (Sheridan, 1980). Even though

automation can improve performance of routine

operations by reducing the workload and reducing

human error at behavioural level, it also introduces a

variety of safety critical issues due to lapses in

attention and errors of cognition. Therefore,

vigilance became crucial component of human

performance in many working environments where

automated systems are ubiquitous (Warm et al.,

2008), because lapses in attention and errors

committed by the operator in industry can have

catastrophic consequences.

Vigilance is a term that has been used in

different ways by different scientist and therefore, it

has various definitions (Oken et al., 2006). However,

in cognitive neuroscience and psychology, the term

PsychophysiologicalMeasurementsinRealWorkingEnvironments-WirelessEEGStudyoftheOperators'Vigilance

of vigilance is used to refer ability of organisms to

maintain their focus of attention and to remain alert

over prolonged period of time (Warm et al., 2008).

When studying vigilance, the neuroscientist often

specifically refers to a vigilance decrement and

lapses in the sustained attention over extended

period of time, especially when repetitious,

monotonous and continuous tasks are carried out

(Oken et al., 2006; Warm et al., 2008). However,

many studies of vigilance have shown that for most

of operators engaged in attention and monotonous

tasks, it is not possible to retain a constant level of

alertness (Yu et al., 2007).

Various authors studied the physiological

correlates of vigilance decrement. Hung et al. (2001)

quantified alertness level by correct rates on auditory

and visual vigilance task in an EEG study.

According to them, in the auditory vigilance task,

the relative spectral amplitudes in the alpha and

theta bands as well as the mean frequency spectrum

was found to be the best combination for predicting

the alertness level. In visual vigilance task, the beta

frequency band was the only feature for predicting

alertness level (Huang et al., 2001). This was

supported by the work of the Dockree et al. (2007).

They confirmed that the subjects with higher tonic

frequencies in the alpha range show a larger-

amplitude late positive event-related potential (ERP)

component that has previously been found to predict

a good sustained attention performance. Bonnefond

et al. (2010) showed in an ERP study that vigilance

decline that is reflected by significant changes in

performance and spectral power, is also

accompanied by specific effect of time on the P2 and

Late Positive LP1 component.

Jung et al. (1997) proposed a method for

estimating operators’ level of attention in near real

time, by merging power spectrum estimation,

principal component analysis and artificial neural

networks. They confirmed that there is a close

relation between changes in performance and in

EEG power spectrum.

Pattyn et al. (2008) studied the mechanisms of

sustained attention by targeting two modes of

attention control, endogenous and exogenous

attention. In their study, they did not use the EEG

measurements, instead the reaction times (RT), error

rates (ERs) and heart rate variability (HRV) were

measured and it was found that performance

decrement in the visual vigilance task appears after

approximately 20-30 minutes.

Above mentioned studies almost exclusively

agree on obtaining the information about vigilance

from EEG signals. However, the optimal set of

features for detecting it, as well as their applicability

to on-line monitoring is still out of reach. Identifying

measures that are intuitive, reliable and applicable

for working environment would bring a huge

benefit. These would then serve to tune the future

industrial algorithms.

3.2 Problems regarding Real-life

Vigilance Monitoring

As already stated before, continuous monitoring of

the operators’ mental state, in operational

environment, could decrease potential for serious

errors and provide valuable information concerning

the ergonomics of the tasks being performed (Gevins

et al., 1995). However, in order to achieve this kind

of monitoring, specific requirements need to be

fulfilled including the reliable measurements using

inexpensive and highly mobile equipment. These are

the necessary prerequisites for conducting brain

activity monitoring outside the laboratory settings

(Gevins et al., 1995). Therefore, the non-invasive,

small in size and weight, and relatively inexpensive

wireless EEG device that produces reliable results,

seems like a promising solution for the above-

mentioned problems. Even though mobile EEG

represents a promising tool for the experiments to be

carried outside the laboratory settings, it is far from

being the only requirement for achievement of

reliable psychophysiological results. In fact, it is

necessary to carefully design and set up the

experiment itself due to limitations of use of

presentation software and other conditions

contributing to realistic character of the improvised

workplace.

Previously mentioned studies were conducted in

the strictly controlled laboratory conditions using

presentation software and obtaining the time-locked

events in order to study, mostly, the ERPs and time

locked power spectral densities. Further, apart from

measuring the electrocortical brain activity, in most

of these studies, the reaction time (to a stimuli) was

one of the main parameters for measuring the

vigilance level. At this point, it is important to

outline the difficulty of obtaining the time-locked

events in dynamic real working environment. We

believe that if there is a requirement for operator to

put additional effort for the button press, or any

other action in order to obtain the time-locked events

while performing actual task, the credibility of

results will decrease as in that case the working

environment is changing as the action that operator

is carrying out is altered.

Therefore, one of the biggest challenges

BIOSTEC2014-DoctoralConsortium

regarding EEG recording outside the laboratory

settings, assuming the movement related artefacts

already being suppressed, is to find a way to obtain

the time locking events, without distracting the

operator from performing his task in order to asses

the information regarding the ERP components and

time-locked spectral densities.

4 STATE OF THE ART

Recent studies indicated that it is possible to

measure the human state of vigilance using EEG

signals (Oken et al., 2006; Yu et al., 2007; Dockree

et al., 2007; Huang et al., 2007; Bonnefond et al.,

2010). It has been further reported that it is possible

to estimate the vigilance level in near real-time

(Jung et al., 1997). However, all of the mentioned

studies have one common limitation, that is, all of

them have been performed in laboratory conditions,

with constricted movements and featuring bulky

equipment.

Apart from the advantage that wireless EEG

systems are small enough and allows the on-field

measurement of the operators’ vigilance state, this

setting also allows to the operator to move naturally

in the working environment, without causing huge

instrumentation artefacts. This is supported by the

fact that movement of electrode wires, a major

source of instrumentation artefacts (Usakli, 2010),

and electrode displacement are minimized and

therefore, artefacts that are not physiologically

related are suppressed (Debener et al., 2012).

Recently a few EEG studies were conducted in

various areas, including the study of vigilance.

However, the most of these studies were carried out

in order to assess information regarding the drivers

safety and safety of the pilots in aviation industry

and not for operators that are working in industry

sector. This is not surprising, since the most of the

advances in HFE, including the early beginnings of

this field, are coming from aviation industry (Canas

et al., 2011). On the other hand, a large portion of

the research in the field of mobile EEG is conducted

in the scope of medical studies, example being the

work of Lin et al. (2008) in which non-invasive

neural prostheses was proposed for continuously

monitoring high-temporal resolution brain dynamics

using wireless EEG.

It is important to note, that while carrying out the

above-mentioned studies for driver and/or pilot

vigilance, virtual reality software was used. While it

allowed replicating the real-life situations to a large

extent, it still could not sufficiently mimic the real

environment and all of ifs features.

Further, the majority of the previous studies were

carried out using commercial wireless EEG gadgets

with dry and non-contact electrodes that are

becoming increasingly popular, mostly due to

gaming purposes. The attractive side of these

commercial devices lies in their comfortable and

easy-to-use character, e.g. stemming from the fact

that the traditional, gel electrodes are avoided.

However, on the downside, they currently provide

the limited signal strength and therefore, the results

obtained with this equipment are not reliable.

Debener et al. (2012), on the other hand, showed

that it is possible to modify one of the consumer 14-

channel wirelesses EEG in order to improve the

signal quality, obtaining the state-of-the art device

for further research. In the aforementioned study,

wireless and mobile character of the consumer EEG

device was combined with the research-grade

electrodes allowing for high-quality recordings.

5 METHODOLOGY

5.1 Vigilance Monitoring

Measuring EEG in an unrestricted environment

always gathered close attention of scientific

community. However, high-quality, medical graded

recordings were only recently demonstrated with

fully mobile platforms (Debener et al. 2012).

Company mBrainTrain recently provided a research

tool for neurofeedback testing paradigms. We

adapted their "SMARTING" device for vigilance

research task mainly due to two features:

1) Small, lightweight and mobile character, and

2) Close to medical signal quality

Figure 2 portrays a close-up of "SMARTING"

while being prepared for recording. This solution

uses gel-based electrodes produced by the renown

Easycap company (Germany), offering high-quality

recordings mainly due to low impedance.

This system features 24 EEG channels with 24-

bit resolution. The real-time data transmission is

achieved using the Bluetooth 2.1 EDR that is able to

communicate with a PC or Android based

phones/tablets. In addition, electrode impedance

information is continuously sent, together with the

gyroscope readings.

PsychophysiologicalMeasurementsinRealWorkingEnvironments-WirelessEEGStudyoftheOperators'Vigilance

Figure 2: Mounting of "SMARTING": recording

electrodes are filled with conductive gel.

Figure 3 shows a preview from our initial quality

tests. A simulated "lean" factory work place was

also created (Figure 4). Although still in the fine-

tuning stage, it can faithfully replicate several

common working places related to the targeted,

repetitive factory tasks.

Figure 3: Signals with a moving subject exhibited almost

no visual movement artefacts.

5.2 Signal Segmentation

For tracking of changes in vigilance, it is necessary

to properly segment the signal according to the

performed action. This is not an easy task taking into

account the often vague beginning and the end of

this process.

We decided to take a two-step approach to this:

1) Provide a sound signal, identical to the sound

that is produced by the machine used by the

operator at the real working environment, to

mark a beginning of the operation (and also

serve as a trigger signal for the EEG

segmentation), or

2) Provide a signal, not notable for the operator

but coinciding with the beginning of the

operation.

While precise triggering is still a hard task, we

believe that introducing a microphone device that

would serve as a reliable reference can solve this

problem.

The logic behind "notable" and "not notable"

triggers is to eliminate (but also investigate)

vigilance dependency on outside factors like this

one.

Figure 4: Realistic "lean" workplace, where simulated

repetitive task is performed.

5.3 Signal Processing

The use precise triggering will enable the

comparison between spectral EEG measures and the

ERPs obtained after the signal segmentation. The

rationale lies in the fact that ERPs, if proven

sufficiently reliable compared to established spectral

measures, can offer much faster detection of the

decrease in vigilance. In this scope, the changes in

the delay or amplitude of e.g. P300 can be related to

the depth of cognitively processing the stimulus that

further can be related to the level of attention

(Murata et al., 2005).

In parallel, we plan to segment the continuous

EEG signal, according to the duration of one

complete operators’ action, in order to investigate

known frequency components of interest, theta (4-8

Hz), alpha (8-13) and beta (13-30 Hz) bands..

Further, due to the non-stationary nature of EEG

signals and in order to achieve precise time-

frequency resolution, we will use decomposition

techniques including Wavelet transform and

Empirical Mode Decomposition (Huang et al. 1998,

Mijovic et al., 2010). Then, spectral correlates for

vigilance monitoring will be traced and the delay in

their detection used to assess their suitability as a

real-time vigilance-tracking marker.

Temporal analysis of ERP- based vigilance

detectors would be achieved by first training a

classifier in case of the present audio stimuli.

BIOSTEC2014-DoctoralConsortium

Response time delays would most likely be used as a

measure of decreased attention. Introducing another

triggering system (e.g. precise enough, yet outside of

the intentional human notice) would be used to

assess how vigilance changes in presence/absence of

distraction.

Also, training ERP and spectral methods for

robotic control is going to take place as well to

pinpoint the usefulness of industry BCI devices.

5.4 Real-time Implementation and

Industry Solution Roadmap

Once the algorithms are developed to positively

detect vigilance shift, they are going to be

implemented online. Following this, the complete

system is expected to be tested within the facilities

of our partners, FIAT Serbia and Tetrapak Serbia.

Further, the minimal viable EEG system to

perform these tasks can be identified from here. This

means fewer sensors, on much fewer previously

identified locations. This would serve as a guideline

for future industry vigilance monitoring system.

6 EXPECTED OUTCOME

The proposed aim of this research is to pinpoint to

routines used to attain close to constant vigilance

level of operators performing industrial, repetitive

tasks.

This is done in order to reduce and/or eliminate

potential slips of sustained attention, which is

desirable from both economic (production

efficiency) as well as health related (work injuries)

points of view.

We expect to deliver real-time algorithms for

vigilance monitoring and notification from one hand,

as well as guidelines for improvement of work

routines taking into account the newly available

vigilance monitoring data from the other.

We will use a high-density

electroencephalographic (EEG) sensor to identify

the limits of necessary "resolution" for vigilance

monitoring. We expect to, following the success of

this task, propose an industrial system that would

reduce the costs and increase the work safety, while

being of acceptable price and comfortable to use.

Attaining the individual privacy, we expect it

will be possible to provide the operator with the

information when his alertness level starts to

decrease, which in turn can yield fewer errors

committed and consequently, increase the overall

industrial safety at the workplace and decrease the

economical losses.

In order to achieve the objectives of this work,

we will firstly conduct psycho-physiological

measurements at the improvised, but authentically

replicated workplace using the novel wireless EEG

system. We will then bring these measurements to

real factory conditions due to collaboration with our

partner companies interested in this concept.

ACKNOWLEDGEMENTS

The presented study is fully funded by the European

project framework FP7 InnHF.

REFERENCES

Bonnefond, A., Doignon, C. N., Touzalin, C. P., and

Dufour, A., 2010. Vigilance and intrinsic maintenance

of alert state: An ERP study. Behavioural brain

research, 211(2), 185-190.

Cacciabue, P. C., 2004. Human error risk management for

engineering systems: a methodology for design, safety

assessment, accident investigation and training.

Reliability Engineering & System Safety, 83(2), 229-

240.

Canas, J. J., Velichovsky, B. B., Velichovsky, B. M.,

2011. Book chapter “Human Factors and

Ergonomics”, in Martin, P. R., Cheung, F. M.,

Knowles, M. C., Kyrios, M., Littlefield, L., Overmier,

J. B., &Prieto, J. M. (Eds.), 2011. IAAP handbook of

applied psychology.Wiley.com.

Debener, S., Minow, F., Emkes, R., Gandras, K., and Vos,

M., 2012.How about taking a lowcost, small, and

wireless EEG for a walk?.Psychophysiology, 49(11),

1617-1621.

Dockree, P. M., Kelly, S. P., Foxe, J. J., Reilly, R. B., and

Robertson, I. H. 2007. Optimal sustained attention is

linked to the spectral content of background EEG

activity: greater ongoing tonic alpha ( 10 Hz) power

supports successful phasic goal activation. European

Journal of Neuroscience, 25(3), 900-907.

Gevins, A., Leong, H., Du, R., Smith, M. E., Le, J.,

DuRousseau, D., Zhang, J., and Libove, J.

1995.Towards measurement of brain function in

operational environments. Biological Psychology,

40(1), 169-186.

Huang, N. E., Shen, Z., Long, S. R., Wu, M. C., Shih, H.

H., Zheng, Q., Yen, N. C., Tung, C. C., and Liu, H. H.,

1998.The empirical mode decomposition and the

Hilbert spectrum for nonlinear and non-stationary

time series analysis. Proceedings of the Royal Society

of London. Series A: Mathematical, Physical and

Engineering Sciences, 454(1971), 903-995.

PsychophysiologicalMeasurementsinRealWorkingEnvironments-WirelessEEGStudyoftheOperators'Vigilance

Huang, R. S., Tsai, L. L., and Kuo, C. J., 2001. Selection

of valid and reliable EEG features for predicting

auditory and visual alertness levels. Proceedings-

national science council republic of china part b life

sciences, 25(1), 17-25.

Jung, T. P., Makeig, S., Stensmo, M., and Sejnowski, T. J.

1997. Estimating alertness from the EEG power

spectrum. Biomedical Engineering, IEEE

Transactions on, 44(1), 60-69.

Lin, C. T., Ko, L. W., Chiou, J. C., Duann, J. R., Huang,

R. S., Liang, S. F., Chiu, T.W., and Jung, T.P. 2008.

Noninvasive neural prostheses using mobile and

wireless EEG. Proceedings of the IEEE, 96(7), 1167-

1183.

Mijovic, B., De Vos, M., Gligorijevic, I., Taelman, J., and

Van Huffel, S. 2010. Source separation from single-

channel recordings by combining empirical-mode

decomposition and independent component analysis.

Biomedical Engineering, IEEE Transactions,57(9),

2188-2196.

Murata, A., Uetake, A., Takasawa, Y., 2005. Evaluation of

mental fatigue using feature parameter extracted from

event-related potential”, International Journal of

Industrial Ergonomics, 35(8),,761–770.

Oken, B. S., Salinsky, M. C., and Elsas, S. M., 2006.

Vigilance, alertness, or sustained attention:

physiological basis and measurement. Clinical

Neurophysiology, 117(9), 1885-1901.

Parasuraman, R., 2003.Neuroergonomics: Research and

practice. Theoretical issues in ergonomics science, 4,

5-20.

Parasuraman, R., Rizzo, M., 2007.Neuroergonomics: The

Brain at Work. Oxford University Press, Oxford.

Pattyn, N., Neyt, X., Henderickx, D., and Soetens, E.,

2008. Psychophysiological investigation of vigilance

decrement: Boredom or cognitive

fatigue?.Physiology&Behavior,93(1), 369-378.

Reason, J. T., 1990. Human Error. Cambridge University

Press, Cambtidge.

Sheridan, T., 1980. ‘Supervisory Control’ in Handbook of

Human Factors, ed G Salvendy, Willey, New York,

pp. 1243-1268.

Usakli, A. B., 2010. Improvement of EEG signal

acquisition: An electrical aspect for state of the art of

front end. Computational intelligence and

neuroscience, volume (2010), article ID 630649.

Warm, J.S., Parasuraman, R., and Matthews, G.,

2008.Vigilance requires hard mental work and is

stressful. Human Factors: The Journal of the Human

Factors and Ergonomics Society, 50(3), 433-441.

Wiegmann, D. A., and Shappell, S. A., 2001. Human error

analysis of commercial aviation accidents:

Application of the Human Factors Analysis and

Classification System (HFACS). Aviation, Space, and

Environmental Medicine, 72 (11), 1006-1016.

Yu, H., Shi, L. C., and Lu, B. L., 2007. Vigilance

Estimation Based on EEG Signals. In Proceedings of

the International Conference on Mechanical

Engineering 2007 (ICME2007).

BIOSTEC2014-DoctoralConsortium