A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN

SIGNALS OR EYE BLINKS

Andr´e Ferreira, Daniel Cruz Cavalieri, Rafael Leal Silva

Teodiano Freire Bastos-Filho and M´ario Sarcinelli-Filho

Department of Electrical Engineering, Federal University of Espirito Santo

Av. Fernando Ferrari, 514, 29075-910, Vit´oria-ES, Brazil

Keywords:

Wheelchair control, Human Machine Interfaces (HMI), Biomedical Signal Processing.

Abstract:

A system allowing a person with severe neuromotor disfunction to choose symbols in a Personal Digital

Assistent (PDA) using electroencephalography (EEG) or electromyography (EMG) is implemented onboard

an electrical wheelchair. Through this system the user is able to elicit personal needs or states, like sleep,

thirst or hunger; to write texts using an alpha-numeric keyboard and to command a robotic wheelchair. The

EEG patterns used are event-related synchronization and de-synchronization (ERS and ERD, respectively)

occurring in the alpha band of the signal spectrum captured in the occipital region of the head, while the

EMG patterns are eye-blinks. The results so far obtained with the system developed, in indoor and outdoor

environments, are quite satisfactory. This paper describes the system so far implemented and shows some

experimental results associated to it.

1 INTRODUCTION

The use of biological signals intentionally generated

by impaired people can contribute to improve their

life-quality, providing augmentative communication

capabilities and autonomy of movement (Wolpaw

et al., 2002; Mill´an et al., 2003). The development of

a Human Machine Interface (HMI) that considers my-

oelectric (EMG) or electroencephalographic (EEG)

signals is here described. Such an HMI acquires the

myoelectric or electroencephalographic signals of an

impaired individual in order to recognizea short set of

easily voluntarily generated patterns, which are asso-

ciated to a group of previously deﬁned tasks. Such an

interface has been used in connection to robotic de-

vices (Ferreira et al., 2006; Frizera-Neto et al., 2006),

and is currently being used to allow an individual to

control a robotic wheelchair and to communicate with

other people, as it is shown hereinafter.

To the extent of the authors’ knowledge, just two

works using EEG to command a wheelchair have

been published so far (Tanaka et al., 2005; Rebsamen

et al., 2007). In (Tanaka et al., 2005) the process-

ing unit is off board the wheelchair, a high number

of EEG electrodes (thirteen) is used, and the recog-

nition rate associated to the brain signal may be as

low as 20%. In (Rebsamen et al., 2007) the focus is

the navigation of the wheelchair (no communication

support is included), and a high number of electrodes

(ﬁfteen) is used. The system here developed, by its

turn, uses only three EEG electrodes, is quite easy

to use, presents a high recognition rate, and is more

versatile, since it allows selecting between two com-

munication channels (EEG or EMG). Besides allow-

ing commanding the wheelchair, the system here de-

scribed provides other useful functions, as it is shown

in Section 4. It was tested in indoor and outdoor en-

vironments, with quite satisfactory results.

The current structure of the proposed HMI and the

way it interacts with the impaired individual and the

wheelchair are shown in Figure 1. The signal ac-

quired by the electrodes connected to the face/head of

the impaired individual are conditioned and quantized

through a high-resolution A/D converter. After being

read by a computer, such a signal is ﬁltered through

a bandpass digital ﬁlter whose pass band spans from

1 to 30 Hz, when using the EMG option, or from

8 to 13 Hz, when acquiring EEG signal (the alpha

band). Features of interest extracted from such sig-

nals are then delivered to a classiﬁer that identiﬁes

if the impaired individual wishes or not to select a

symbol shown on the screen of the Personal Digital

Assistant (PDA) as illustrated in Figure 1. If yes,

the communication interface shown in the ﬁgure asks

the PDA for the information necessary and sends it

to the next module, which is responsible for generat-

Ferreira A., Cruz Cavalieri D., Leal Silva R., Freire Bastos-Filho T. and Sarcinelli-Filho M. (2008).

A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS.

In Proceedings of the First International Conference on Biomedical Electronics and Devices, pages 62-67

DOI: 10.5220/0001048200620067

 SciTePress

Figure 1: The structure of the proposed HMI.

ing the necessary control actions so that the robotic

wheelchair executes the task the impaired individual

has chosen. The feedback is closed through the oper-

ator (bio-feedback), as shown in the ﬁgure.

The description of the system so far implemented

is completed in Section 2. The data acquisition sys-

tem and the techniques used in the EMG and EEG

processing module are presented in Section 3. In the

sequel, a description of the functionalities included

in the graphic interface programmed in the PDA is

presented in Section 4. The experimental results are

presented in Section 5 and are discussed in Section 6,

which highlights the conclusions of the work as well.

2 HARDWARE STRUCTURE

Figure 2 illustrates the hardware embedded on the

wheelchair. Encoders and ultrasonic sensors are func-

tional, while the others are under development. The

motors are controlled by a low-level controller pro-

grammed on a MSP430 microcontroller (Texas In-

struments, Inc.), which receives commands of angu-

lar and linear velocities from a mini-PC onboard the

wheelchair. The signal acquisition system and the

PDA are connected to the PC through a parallel and a

serial port, respectively.

Two electronic boards, a signal conditioning one and

a quantization one, compound the signal-acquisition

system. The signal acquisition board has two input

channels and a third electrode used as the reference

for the signal ampliﬁer. A high-pass ﬁlter with cut-

off frequency of 0.1 Hz avoids the saturation of the

ampliﬁers due to the continuous voltage caused by

the coupling between the electrodes and the skin. A

fourth order low-pass Butterworth ﬁlter with cutoff

frequency of 32 Hz limits the spectrum of the ac-

quired signal and attenuates 60 Hz artifacts (electro-

magnetic induction), some contaminating noise and

disturbances generated by muscles movements, elec-

trodes displacement, etc. Such a board also embeds

a Body Driver circuit to reduce 60 Hz artifacts (Web-

ster, 1998).

The second part of the acquisition system is a

quantization board based on the AD7716 analog to

digital converter. The main features of such a chip

are a resolution correspondent to 22 bits, four A/D

channels, and a low-pass digital ﬁlter with a cutoff

frequency selectable among 36.5 Hz, 73 Hz, 146 Hz,

292 Hz and 584 Hz. The sampling rate used for the

EEG signal is 140 Hz, so that the cutoff frequency of

such a low-pass ﬁlter has been set to 36.5 Hz. The

digital signal thus obtained is then sent to the high

level hardware.

After receiving the acquired data delivered by the

data acquisition board, the onboard CPU (mini-ITX)

is responsible for pre-processing them, extracting the

desired features, classifying them and generating the

control signals associated to the recognized pattern.

Figure 2 illustrates how the hardware pieces are con-

nected.

Figure 2: The hardware structure of the system developed.

A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS

3 RECOGNIZING SIGNAL

PATTERNS

3.1 Through Processing EEG

Event-related Synchronization and De-

synchronization (ERS and ERD, respectively)

are the EEG patterns searched for in this work.

They are characterized by meaningful changes in the

signal energy in speciﬁc frequency bands. An energy

increase is associated to an ERS, while an energy

decrease is associated to an ERD (Pfurtscheller and

da Silva, 1999). The frequency band used to detect

these patterns is the alpha band (8-13 Hz) and, thus,

the digitized signal is ﬁltered by a FIR ﬁlter with

such a pass band.

The EEG signal is acquired in the occipital region

of the user’s head, with electrodes in the positions O

and O

(regarding the 10-20 International System of

Figure 3) and the reference in the right ear.

Figure 3: The 10-20 International System.

A user whose eyes are open (under visual stimulus or

concentrated) keeps the alpha rhythm in a low energy

level. When he/she closes the eyes (with no visual

stimulus or relaxed), there is a great energy increase

in the alpha rhythm, characterizing an ERS. The vari-

ance of the ﬁltered EEG signal allows detecting these

energy changes, as observed in Figure 4.

The second graphic in Figure 4 is generated re-

garding a moving window ﬁlled with N = 280 sam-

ples (N is empirically determined) of the ﬁltered

EEG signal (x

), for which the mean value and

the variance are, respectively, µ =

∑

k=1

and

∑

k=1

− µ)

The variance thus obtained is the input of a

threshold-based classiﬁer, whose function is to detect

if the user wishes or not to select a symbol presented

in the PDA screen. Case yes, a request is sent to the

PDA through a serial line, and it informs which sym-

bol has been selected. Knowing that, the mini-ITX

calculates the necessary control signals to accomplish

Figure 4: (a) Filtered EEG signal with ERD and ERS. (b)

Variance increase during an ERS.

the chosen task and sends it to the actuation module

(MSP430).

3.2 Through Processing EMG

The objective here is to recognize the presence of

an eye-blink (Figure 5) in the Myoelectrical signal

(MES) acquired on the user’s face, for selecting sym-

bols presented in the PDA screen.

The samples of a given MES can be considered

as a random variable, whose variance represents an

averaged measure of the variability or activity of the

signal about its mean (Rangaraj, 2002), as shown in

Figure 6. Such indicator of signal activity was used

to control the robotic wheelchair with good results,

as it is indicated in Table 1. Four individuals tested

the capability of choosing an icon in the PDA screen

through the variance associated to the MES signal.

They should blink an eye to select an icon they were

asked to select. As shown in Table 1, the accuracy

obtained by using the MES variance as the indica-

tor of activity was very high, thus justifying to use

Figure 5: Myoelectrical signal associated to eye-blinks.

BIODEVICES 2008 - International Conference on Biomedical Electronics and Devices

Figure 6: (a) MES of an ordinary individual (male, 25

years). (b) Signal variance.

such an indicator. Each individual was asked to se-

lect 15 icons, each one associated to a movement of

the wheelchair. The calculation of the variance of

the MES is performed in the same way used for the

EEG signal (Subsection 3.1), after normalizing the

signal amplitude. A threshold-based classiﬁer was

used again.

However, according to (Kreifeldt, 1974) the myo-

electrical signal is better described as a stochastic ran-

dom process, for its average and variance vary along

time. Thus, it is necessary to use more robust sys-

tems to process MES, to take into account its stochas-

tic behavior. In (Hudgins et al., 1993) it is shown

that such signal presents a deterministic characteris-

tic in the ﬁrst 200 ms after a muscular contraction.

Due to the nature of the MES, it is reasonable to ex-

pect meaningful changes in the value of the parameter

describing a particular pattern from one individual to

other. Other aspects, like changes in the position of

the electrodes and body-weight ﬂuctuations, will pro-

duce changes in the signal patterns along time (Hud-

gins et al., 1993). Thus, a classiﬁer based on an artiﬁ-

cial neural network (ANN) was trained to accommo-

date the expected individual differences and, as well,

to accept slow variations in the values associated to

the patterns to be recognized.

The neural network is trained using three back-

propagation algorithms: Bayesian Regularization

(BR), Resilient Back-propagation (RP) and Scaled

Conjugate Gradient (SCG) (MathWorks, 2000). A

Table 1: Results for controlling the wheelchair with basis

on the indicator of activity.

Individual # of Tests # of Errors Rightness

A 15 0 100 %

B 15 0 100 %

C 15 1 93.3 %

D 15 0 100 %

Table 2: The ANN’s implemented and the training algo-

rithms.

Training Input Hidden Output Error

Algorithm Layer Layer Layer (%)

BR 20 4 3 0.6

BR 20 6 3 0.5

BR 20 8 3 0.6

BR 20 10 3 0.6

RP 20 4 3 0.3

RP 20 6 3 0.6

RP 20 8 3 0.3

RP 20 10 3 0.5

SCG 20 4 3 0.8

SCG 20 6 3 0.5

SCG 20 8 3 0.3

SCG 20 10 3 0.3

total of 210 pre-processed sequences of facial MES

correspondent to right-eye blinks, 210 pre-processed

samples of facial MES correspondent to left-eye

blinks, and 210 sequences of random noise, result-

ing in 630 sequences of signal samples, were used for

training and validating the neural networks tested. For

each one of these three sets of samples, ﬁfty percent

were used for training and ﬁfty percent were used for

validation. The results are shown in Table 2. A hid-

den layer with 4 to 10 neurons resulted in a very good

accuracy in classifying the three patterns of interest,

knowing left-eye blink, right-eye blink and noise (no

blink). The error presented in Table 2 is the sum of

the errors during training and validation.

The ANN conﬁgurations showing the best perfor-

mances in Table 2 were tested, now regarding 252

new test signals, from which 84 corresponds to left-

eye blinks, 84 corresponds to right-eye blinks and 84

corresponds to noise sequences (no blinks). The re-

sults obtained when classifying them are presented in

Table 3, and show that the use of the artiﬁcial neural

network as a classiﬁer for the patterns searched for

in the MES resulted in a high rate of rightness. In

particular, the classiﬁer currently implemented in the

system here addressed is an artiﬁcial neural network

having 20 neurons in the input layer, 4 neurons in the

hidden layer and 3 neurons in the output layer, whose

training algorithm is the Resilient Back-propagation.

Table 3: Results of testing the best ANN’s in Table 2.

Training # of Neurons in the Success

Algorithm Hidden Layer Rate (%)

BR 6 98.4

RP 4 99.6

SCG 10 98.4

A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS

4 SELECTING AN OPTION ON

THE PDA SCREEN

The PDA (Figure 1) is installed onboard the

wheelchair in such a way that it is always visible for

the impaired individual seated on it. It provides a

graphic interface containing the possible options for

the operator, including the pre-programmed move-

ments of the wheelchair, a virtual keyboard for text

edition, and symbols to express some basic needs or

feelings of the impaired individual, such as to sleep,

drink, eat, feel cold, heat, etc. For all these cases, a

speciﬁc option is selected using a procedure to scan

the rows and columns in which the icons are dis-

tributed on the PDA screen (once the desired screen

is presented). A voice player conﬁrms the option cho-

sen, providing a feedback to the user and allowing

the communication with people around as well, either

through EMG or through EEG signals.

The operator selects symbols presented on the

PDA screen, which are distributed in a form that re-

sembles a matrix, assisted by an automatic scanning

system. Each row of the matrix of symbols remains

pre-selected for a while, until the operator conﬁrms

the choice. After selecting the row, the process is re-

peated, now regarding the columns of that row. For

tasks like controlling the wheelchair or asking for spe-

ciﬁc external help, this scanning system is quite suit-

able.

Figure 7 illustrates the

STATE

and

TEXT

screens.

The ﬁrst one is designed to support interpersonal

communication. It presents options to the operator in

order to elicit emotions, personal states or some basic

needs. Although the options of this screen can be ex-

pressed by using the

TEXT

screen, this mode is much

faster, mainly in emergencies, such as to complain

about pain. The

TEXT

mode provides a communica-

tion channel to the operator, allowing the selection of

letters and numbersthrough a virtual keyboard, whose

sounds are echoed to speakers. Although being a low

bit-rate communication process, it provides a way to

elicit words via artiﬁcial voice when the patient does

not have this capacity anymore.

The screen

MOVEMENT

provides to the operator a

set of symbols corresponding to movements of the

robotic wheelchair. The options are shown in Fig-

ure 8, and represents actions sent directly to the

wheelchair motors. The ﬁrst command starts the

movement of the wheelchair and the next one, it

does not mind where the automatic scan is, stops

the wheelchair. For safety, only successive short

back-displacements are allowed, because of the null

visibility in such a movement. Through the screen

CONTROLLER

, option currently being developed, the

Figure 7: Software

RWCC

STATE.

and

TEXT

screens.

Figure 8: Software

RWCC

MOVEMENT.

and

CONTROLLER

screens.

user will be able to choose a place in a structured en-

vironment, and the wheelchair will be guided to that

place by an automatic control system. This screen is

shown in Figure 8.

5 EXPERIMENTS

A user testing the wheelchair in an indoor environ-

ment (left) and in an outdoor environment (right), us-

ing the EEG signal option of the developed HMI, is

presented in Figure 9. The preparation of the user

to operate the system consists of cleaning the regions

where the electrodes should be connected (the O

and

positions in the head and the right earlobe) and,

then, a special gel is applied between the electrode

and the user’s skin, for impedance-matching.

A meaningful group of users tested their capabil-

ity of using the system, and the result was that all of

them were capable to command the wheelchair and to

communicate with people around it.

The analysis of the EEG signal in the alpha band,

and of its variance as well, shows very clearly when

BIODEVICES 2008 - International Conference on Biomedical Electronics and Devices

Figure 9: Testing the system prototype.

the user closes his/her eyes, generating an ERS (the

signal variance exhibits a great increase, as shown

in Figure 4). This indicates that the user wishes to

select the option currently highlighted in the PDA

screen. An important aspect, regarding the detection

of the changes in the signal variance, is that an ad-

justable hysteresis-zone is included in the threshold-

based classiﬁer in order to increase the system robust-

ness, thus avoiding false ERS/ERD detection.

6 CONCLUSIONS

The HMI so far developed was tested in indoor and

outdoor environments, with quite satisfactory results,

according to the statements of the users who operated

the prototype during the tests.

The acquisition system allied to the PDA has

proven to be quite efﬁcient for choosing commands

to the wheelchair using EMG or EEG signals. A min-

imum knowledge about the HMI and a very quick

training is required to operate the whole system.

However, it is worthy to mention that so far the

developed HMI has not been tested by people with

severe neuromotor disabilities, which is the next step

of this work.

The ANN used in the analysis of MES has demon-

strated a very good capability to ﬁnd the desired pat-

terns in such signals. The feedforward topology with

back-propagation training algorithm, having two ac-

tive and one hidden layers, allowed a satisfactory rate

of classiﬁcation rightness.

The easiness of electrode-placing (for both EMG

and EEG options), the simplicity of the graphical in-

terface running in the PDA and the easiness to adapt

the system to a commercial electrical wheelchair are

the major advantages of the HMI here developed,

when taking into account the ﬁnal users of this as-

sistive technology.

ACKNOWLEDGEMENTS

The authors thank CAPES, a foundation of the

Brazilian Ministry of Education (Project 150(07)),

FAPES, a foundation of the Secretary of Science and

Technology of the State of Espirito Santo (Process

30897440/2005), and FACITEC/PMV, a fund of the

Vitoria City Hall for supporting Scientiﬁc and Tech-

nological Development, for their ﬁnancial support to

this research.

REFERENCES

Ferreira, A., Bastos-Filho, T. F., Sarcinelli-Filho, M.,

Cheein, F. A., Postigo, J. F., and Carelli, R. (2006).

Teleoperation of an industrial manipulator through a

TCP/IP channel using EEG signals. In Proceed-

ings of the International Symposium on Industrial

Electronics (ISIE 2006), pages 3066–3071, Montreal,

Canada.

Frizera-Neto, A., Celeste, W. C., Martins, V. R., Bastos-

Filho, T. F., and Sarcinelli-Filho, M. (2006). Human-

machine interface based on electro-biological signals

for mobile vehicles. In Proceedings of the Inter-

national Symposium on Industrial Electronics (ISIE

2006), pages 2954–2959, Montreal, Canada.

Hudgins, B., Parker, P., and Scott, R. N. (1993). A new

strategy for multifunction myoelectric control. IEEE

Transactions on Biomedical Engineering, 40:82–94.

Kreifeldt, J. G. (1974). A signal-to-noise investigation of

nonlinear electromyography processors. IEEE Trans-

actions on Biomedical Engineering, 21:298–308.

MathWorks (2000). Neural network toolbox users guide.

Mill´an, J., Renkens, F., Mouri˜no, J., and Gerstner, W.

(2003). Non-invasive brain-actuated control of a mo-

bile robot. In Proceedings of the 18th International

Joint Conference on Artiﬁcial Intelligence, Acapulco,

M´exico.

Pfurtscheller, G. and da Silva, F. H. L. (1999). Event-

related eeg/meg synchronization and desynchroniza-

tion: basic principles. Clinical Neurophysiology,

110(11):1842–1857.

Rangaraj, M. R. (2002). Biomedical Signal Analysis. Wiley

Interscience, New York.

Rebsamen, B., Burdet, E., Guan, C., Zhang, H., Teo, C. L.,

Zeng, Q., Laugier, C., and Ang Jr., M. H. (2007). Con-

trolling a wheelchair indoors using thought. IEEE In-

telligent Systems, 22:18–24.

Tanaka, K., Matsunaga, K., and Wang, H. (2005).

Electroencephalogram-based control of an electric

wheelchair. IEEE Transactions on Robotics, 21:762–

766.

Webster, J. G. (1998). Medical Instrumentation. Applica-

tion and Design. John Wiley & Sons, Inc., New York,

3 edition.

Wolpaw, J. R., Birbaumer, N., McFarland, D. J.,

Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-

computer interfaces for communication and control.

Clin Neurophysiol, 113(6):767–791.

A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS