A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN
SIGNALS OR EYE BLINKS
Andr´e Ferreira, Daniel Cruz Cavalieri, Rafael Leal Silva
Teodiano Freire Bastos-Filho and M´ario Sarcinelli-Filho
Department of Electrical Engineering, Federal University of Espirito Santo
Av. Fernando Ferrari, 514, 29075-910, Vit´oria-ES, Brazil
Keywords:
Wheelchair control, Human Machine Interfaces (HMI), Biomedical Signal Processing.
Abstract:
A system allowing a person with severe neuromotor disfunction to choose symbols in a Personal Digital
Assistent (PDA) using electroencephalography (EEG) or electromyography (EMG) is implemented onboard
an electrical wheelchair. Through this system the user is able to elicit personal needs or states, like sleep,
thirst or hunger; to write texts using an alpha-numeric keyboard and to command a robotic wheelchair. The
EEG patterns used are event-related synchronization and de-synchronization (ERS and ERD, respectively)
occurring in the alpha band of the signal spectrum captured in the occipital region of the head, while the
EMG patterns are eye-blinks. The results so far obtained with the system developed, in indoor and outdoor
environments, are quite satisfactory. This paper describes the system so far implemented and shows some
experimental results associated to it.
1 INTRODUCTION
The use of biological signals intentionally generated
by impaired people can contribute to improve their
life-quality, providing augmentative communication
capabilities and autonomy of movement (Wolpaw
et al., 2002; Mill´an et al., 2003). The development of
a Human Machine Interface (HMI) that considers my-
oelectric (EMG) or electroencephalographic (EEG)
signals is here described. Such an HMI acquires the
myoelectric or electroencephalographic signals of an
impaired individual in order to recognizea short set of
easily voluntarily generated patterns, which are asso-
ciated to a group of previously defined tasks. Such an
interface has been used in connection to robotic de-
vices (Ferreira et al., 2006; Frizera-Neto et al., 2006),
and is currently being used to allow an individual to
control a robotic wheelchair and to communicate with
other people, as it is shown hereinafter.
To the extent of the authors’ knowledge, just two
works using EEG to command a wheelchair have
been published so far (Tanaka et al., 2005; Rebsamen
et al., 2007). In (Tanaka et al., 2005) the process-
ing unit is off board the wheelchair, a high number
of EEG electrodes (thirteen) is used, and the recog-
nition rate associated to the brain signal may be as
low as 20%. In (Rebsamen et al., 2007) the focus is
the navigation of the wheelchair (no communication
support is included), and a high number of electrodes
(fifteen) is used. The system here developed, by its
turn, uses only three EEG electrodes, is quite easy
to use, presents a high recognition rate, and is more
versatile, since it allows selecting between two com-
munication channels (EEG or EMG). Besides allow-
ing commanding the wheelchair, the system here de-
scribed provides other useful functions, as it is shown
in Section 4. It was tested in indoor and outdoor en-
vironments, with quite satisfactory results.
The current structure of the proposed HMI and the
way it interacts with the impaired individual and the
wheelchair are shown in Figure 1. The signal ac-
quired by the electrodes connected to the face/head of
the impaired individual are conditioned and quantized
through a high-resolution A/D converter. After being
read by a computer, such a signal is filtered through
a bandpass digital filter whose pass band spans from
1 to 30 Hz, when using the EMG option, or from
8 to 13 Hz, when acquiring EEG signal (the alpha
band). Features of interest extracted from such sig-
nals are then delivered to a classifier that identifies
if the impaired individual wishes or not to select a
symbol shown on the screen of the Personal Digital
Assistant (PDA) as illustrated in Figure 1. If yes,
the communication interface shown in the figure asks
the PDA for the information necessary and sends it
to the next module, which is responsible for generat-
62
Ferreira A., Cruz Cavalieri D., Leal Silva R., Freire Bastos-Filho T. and Sarcinelli-Filho M. (2008).
A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS.
In Proceedings of the First International Conference on Biomedical Electronics and Devices, pages 62-67
DOI: 10.5220/0001048200620067
Copyright
c
SciTePress
Figure 1: The structure of the proposed HMI.
ing the necessary control actions so that the robotic
wheelchair executes the task the impaired individual
has chosen. The feedback is closed through the oper-
ator (bio-feedback), as shown in the figure.
The description of the system so far implemented
is completed in Section 2. The data acquisition sys-
tem and the techniques used in the EMG and EEG
processing module are presented in Section 3. In the
sequel, a description of the functionalities included
in the graphic interface programmed in the PDA is
presented in Section 4. The experimental results are
presented in Section 5 and are discussed in Section 6,
which highlights the conclusions of the work as well.
2 HARDWARE STRUCTURE
Figure 2 illustrates the hardware embedded on the
wheelchair. Encoders and ultrasonic sensors are func-
tional, while the others are under development. The
motors are controlled by a low-level controller pro-
grammed on a MSP430 microcontroller (Texas In-
struments, Inc.), which receives commands of angu-
lar and linear velocities from a mini-PC onboard the
wheelchair. The signal acquisition system and the
PDA are connected to the PC through a parallel and a
serial port, respectively.
Two electronic boards, a signal conditioning one and
a quantization one, compound the signal-acquisition
system. The signal acquisition board has two input
channels and a third electrode used as the reference
for the signal amplifier. A high-pass filter with cut-
off frequency of 0.1 Hz avoids the saturation of the
amplifiers due to the continuous voltage caused by
the coupling between the electrodes and the skin. A
fourth order low-pass Butterworth filter with cutoff
frequency of 32 Hz limits the spectrum of the ac-
quired signal and attenuates 60 Hz artifacts (electro-
magnetic induction), some contaminating noise and
disturbances generated by muscles movements, elec-
trodes displacement, etc. Such a board also embeds
a Body Driver circuit to reduce 60 Hz artifacts (Web-
ster, 1998).
The second part of the acquisition system is a
quantization board based on the AD7716 analog to
digital converter. The main features of such a chip
are a resolution correspondent to 22 bits, four A/D
channels, and a low-pass digital filter with a cutoff
frequency selectable among 36.5 Hz, 73 Hz, 146 Hz,
292 Hz and 584 Hz. The sampling rate used for the
EEG signal is 140 Hz, so that the cutoff frequency of
such a low-pass filter has been set to 36.5 Hz. The
digital signal thus obtained is then sent to the high
level hardware.
After receiving the acquired data delivered by the
data acquisition board, the onboard CPU (mini-ITX)
is responsible for pre-processing them, extracting the
desired features, classifying them and generating the
control signals associated to the recognized pattern.
Figure 2 illustrates how the hardware pieces are con-
nected.
Figure 2: The hardware structure of the system developed.
A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS
63
3 RECOGNIZING SIGNAL
PATTERNS
3.1 Through Processing EEG
Event-related Synchronization and De-
synchronization (ERS and ERD, respectively)
are the EEG patterns searched for in this work.
They are characterized by meaningful changes in the
signal energy in specific frequency bands. An energy
increase is associated to an ERS, while an energy
decrease is associated to an ERD (Pfurtscheller and
da Silva, 1999). The frequency band used to detect
these patterns is the alpha band (8-13 Hz) and, thus,
the digitized signal is filtered by a FIR filter with
such a pass band.
The EEG signal is acquired in the occipital region
of the user’s head, with electrodes in the positions O
1
and O
2
(regarding the 10-20 International System of
Figure 3) and the reference in the right ear.
Figure 3: The 10-20 International System.
A user whose eyes are open (under visual stimulus or
concentrated) keeps the alpha rhythm in a low energy
level. When he/she closes the eyes (with no visual
stimulus or relaxed), there is a great energy increase
in the alpha rhythm, characterizing an ERS. The vari-
ance of the filtered EEG signal allows detecting these
energy changes, as observed in Figure 4.
The second graphic in Figure 4 is generated re-
garding a moving window filled with N = 280 sam-
ples (N is empirically determined) of the filtered
EEG signal (x
k
), for which the mean value and
the variance are, respectively, µ =
1
N
N
k=1
x
k
and
σ
2
=
1
N
N
k=1
(x
k
µ)
2
.
The variance thus obtained is the input of a
threshold-based classifier, whose function is to detect
if the user wishes or not to select a symbol presented
in the PDA screen. Case yes, a request is sent to the
PDA through a serial line, and it informs which sym-
bol has been selected. Knowing that, the mini-ITX
calculates the necessary control signals to accomplish
Figure 4: (a) Filtered EEG signal with ERD and ERS. (b)
Variance increase during an ERS.
the chosen task and sends it to the actuation module
(MSP430).
3.2 Through Processing EMG
The objective here is to recognize the presence of
an eye-blink (Figure 5) in the Myoelectrical signal
(MES) acquired on the user’s face, for selecting sym-
bols presented in the PDA screen.
The samples of a given MES can be considered
as a random variable, whose variance represents an
averaged measure of the variability or activity of the
signal about its mean (Rangaraj, 2002), as shown in
Figure 6. Such indicator of signal activity was used
to control the robotic wheelchair with good results,
as it is indicated in Table 1. Four individuals tested
the capability of choosing an icon in the PDA screen
through the variance associated to the MES signal.
They should blink an eye to select an icon they were
asked to select. As shown in Table 1, the accuracy
obtained by using the MES variance as the indica-
tor of activity was very high, thus justifying to use
Figure 5: Myoelectrical signal associated to eye-blinks.
BIODEVICES 2008 - International Conference on Biomedical Electronics and Devices
64
Figure 6: (a) MES of an ordinary individual (male, 25
years). (b) Signal variance.
such an indicator. Each individual was asked to se-
lect 15 icons, each one associated to a movement of
the wheelchair. The calculation of the variance of
the MES is performed in the same way used for the
EEG signal (Subsection 3.1), after normalizing the
signal amplitude. A threshold-based classifier was
used again.
However, according to (Kreifeldt, 1974) the myo-
electrical signal is better described as a stochastic ran-
dom process, for its average and variance vary along
time. Thus, it is necessary to use more robust sys-
tems to process MES, to take into account its stochas-
tic behavior. In (Hudgins et al., 1993) it is shown
that such signal presents a deterministic characteris-
tic in the first 200 ms after a muscular contraction.
Due to the nature of the MES, it is reasonable to ex-
pect meaningful changes in the value of the parameter
describing a particular pattern from one individual to
other. Other aspects, like changes in the position of
the electrodes and body-weight fluctuations, will pro-
duce changes in the signal patterns along time (Hud-
gins et al., 1993). Thus, a classifier based on an artifi-
cial neural network (ANN) was trained to accommo-
date the expected individual differences and, as well,
to accept slow variations in the values associated to
the patterns to be recognized.
The neural network is trained using three back-
propagation algorithms: Bayesian Regularization
(BR), Resilient Back-propagation (RP) and Scaled
Conjugate Gradient (SCG) (MathWorks, 2000). A
Table 1: Results for controlling the wheelchair with basis
on the indicator of activity.
Individual # of Tests # of Errors Rightness
A 15 0 100 %
B 15 0 100 %
C 15 1 93.3 %
D 15 0 100 %
Table 2: The ANN’s implemented and the training algo-
rithms.
Training Input Hidden Output Error
Algorithm Layer Layer Layer (%)
BR 20 4 3 0.6
BR 20 6 3 0.5
BR 20 8 3 0.6
BR 20 10 3 0.6
RP 20 4 3 0.3
RP 20 6 3 0.6
RP 20 8 3 0.3
RP 20 10 3 0.5
SCG 20 4 3 0.8
SCG 20 6 3 0.5
SCG 20 8 3 0.3
SCG 20 10 3 0.3
total of 210 pre-processed sequences of facial MES
correspondent to right-eye blinks, 210 pre-processed
samples of facial MES correspondent to left-eye
blinks, and 210 sequences of random noise, result-
ing in 630 sequences of signal samples, were used for
training and validating the neural networks tested. For
each one of these three sets of samples, fifty percent
were used for training and fifty percent were used for
validation. The results are shown in Table 2. A hid-
den layer with 4 to 10 neurons resulted in a very good
accuracy in classifying the three patterns of interest,
knowing left-eye blink, right-eye blink and noise (no
blink). The error presented in Table 2 is the sum of
the errors during training and validation.
The ANN configurations showing the best perfor-
mances in Table 2 were tested, now regarding 252
new test signals, from which 84 corresponds to left-
eye blinks, 84 corresponds to right-eye blinks and 84
corresponds to noise sequences (no blinks). The re-
sults obtained when classifying them are presented in
Table 3, and show that the use of the artificial neural
network as a classifier for the patterns searched for
in the MES resulted in a high rate of rightness. In
particular, the classifier currently implemented in the
system here addressed is an artificial neural network
having 20 neurons in the input layer, 4 neurons in the
hidden layer and 3 neurons in the output layer, whose
training algorithm is the Resilient Back-propagation.
Table 3: Results of testing the best ANN’s in Table 2.
Training # of Neurons in the Success
Algorithm Hidden Layer Rate (%)
BR 6 98.4
RP 4 99.6
SCG 10 98.4
A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS
65
4 SELECTING AN OPTION ON
THE PDA SCREEN
The PDA (Figure 1) is installed onboard the
wheelchair in such a way that it is always visible for
the impaired individual seated on it. It provides a
graphic interface containing the possible options for
the operator, including the pre-programmed move-
ments of the wheelchair, a virtual keyboard for text
edition, and symbols to express some basic needs or
feelings of the impaired individual, such as to sleep,
drink, eat, feel cold, heat, etc. For all these cases, a
specific option is selected using a procedure to scan
the rows and columns in which the icons are dis-
tributed on the PDA screen (once the desired screen
is presented). A voice player confirms the option cho-
sen, providing a feedback to the user and allowing
the communication with people around as well, either
through EMG or through EEG signals.
The operator selects symbols presented on the
PDA screen, which are distributed in a form that re-
sembles a matrix, assisted by an automatic scanning
system. Each row of the matrix of symbols remains
pre-selected for a while, until the operator confirms
the choice. After selecting the row, the process is re-
peated, now regarding the columns of that row. For
tasks like controlling the wheelchair or asking for spe-
cific external help, this scanning system is quite suit-
able.
Figure 7 illustrates the
STATE
and
TEXT
screens.
The first one is designed to support interpersonal
communication. It presents options to the operator in
order to elicit emotions, personal states or some basic
needs. Although the options of this screen can be ex-
pressed by using the
TEXT
screen, this mode is much
faster, mainly in emergencies, such as to complain
about pain. The
TEXT
mode provides a communica-
tion channel to the operator, allowing the selection of
letters and numbersthrough a virtual keyboard, whose
sounds are echoed to speakers. Although being a low
bit-rate communication process, it provides a way to
elicit words via artificial voice when the patient does
not have this capacity anymore.
The screen
MOVEMENT
provides to the operator a
set of symbols corresponding to movements of the
robotic wheelchair. The options are shown in Fig-
ure 8, and represents actions sent directly to the
wheelchair motors. The first command starts the
movement of the wheelchair and the next one, it
does not mind where the automatic scan is, stops
the wheelchair. For safety, only successive short
back-displacements are allowed, because of the null
visibility in such a movement. Through the screen
CONTROLLER
, option currently being developed, the
Figure 7: Software
RWCC
:
STATE.
and
TEXT
screens.
Figure 8: Software
RWCC
:
MOVEMENT.
and
CONTROLLER
screens.
user will be able to choose a place in a structured en-
vironment, and the wheelchair will be guided to that
place by an automatic control system. This screen is
shown in Figure 8.
5 EXPERIMENTS
A user testing the wheelchair in an indoor environ-
ment (left) and in an outdoor environment (right), us-
ing the EEG signal option of the developed HMI, is
presented in Figure 9. The preparation of the user
to operate the system consists of cleaning the regions
where the electrodes should be connected (the O
1
and
O
2
positions in the head and the right earlobe) and,
then, a special gel is applied between the electrode
and the user’s skin, for impedance-matching.
A meaningful group of users tested their capabil-
ity of using the system, and the result was that all of
them were capable to command the wheelchair and to
communicate with people around it.
The analysis of the EEG signal in the alpha band,
and of its variance as well, shows very clearly when
BIODEVICES 2008 - International Conference on Biomedical Electronics and Devices
66
Figure 9: Testing the system prototype.
the user closes his/her eyes, generating an ERS (the
signal variance exhibits a great increase, as shown
in Figure 4). This indicates that the user wishes to
select the option currently highlighted in the PDA
screen. An important aspect, regarding the detection
of the changes in the signal variance, is that an ad-
justable hysteresis-zone is included in the threshold-
based classifier in order to increase the system robust-
ness, thus avoiding false ERS/ERD detection.
6 CONCLUSIONS
The HMI so far developed was tested in indoor and
outdoor environments, with quite satisfactory results,
according to the statements of the users who operated
the prototype during the tests.
The acquisition system allied to the PDA has
proven to be quite efficient for choosing commands
to the wheelchair using EMG or EEG signals. A min-
imum knowledge about the HMI and a very quick
training is required to operate the whole system.
However, it is worthy to mention that so far the
developed HMI has not been tested by people with
severe neuromotor disabilities, which is the next step
of this work.
The ANN used in the analysis of MES has demon-
strated a very good capability to find the desired pat-
terns in such signals. The feedforward topology with
back-propagation training algorithm, having two ac-
tive and one hidden layers, allowed a satisfactory rate
of classification rightness.
The easiness of electrode-placing (for both EMG
and EEG options), the simplicity of the graphical in-
terface running in the PDA and the easiness to adapt
the system to a commercial electrical wheelchair are
the major advantages of the HMI here developed,
when taking into account the final users of this as-
sistive technology.
ACKNOWLEDGEMENTS
The authors thank CAPES, a foundation of the
Brazilian Ministry of Education (Project 150(07)),
FAPES, a foundation of the Secretary of Science and
Technology of the State of Espirito Santo (Process
30897440/2005), and FACITEC/PMV, a fund of the
Vitoria City Hall for supporting Scientific and Tech-
nological Development, for their financial support to
this research.
REFERENCES
Ferreira, A., Bastos-Filho, T. F., Sarcinelli-Filho, M.,
Cheein, F. A., Postigo, J. F., and Carelli, R. (2006).
Teleoperation of an industrial manipulator through a
TCP/IP channel using EEG signals. In Proceed-
ings of the International Symposium on Industrial
Electronics (ISIE 2006), pages 3066–3071, Montreal,
Canada.
Frizera-Neto, A., Celeste, W. C., Martins, V. R., Bastos-
Filho, T. F., and Sarcinelli-Filho, M. (2006). Human-
machine interface based on electro-biological signals
for mobile vehicles. In Proceedings of the Inter-
national Symposium on Industrial Electronics (ISIE
2006), pages 2954–2959, Montreal, Canada.
Hudgins, B., Parker, P., and Scott, R. N. (1993). A new
strategy for multifunction myoelectric control. IEEE
Transactions on Biomedical Engineering, 40:82–94.
Kreifeldt, J. G. (1974). A signal-to-noise investigation of
nonlinear electromyography processors. IEEE Trans-
actions on Biomedical Engineering, 21:298–308.
MathWorks (2000). Neural network toolbox users guide.
Mill´an, J., Renkens, F., Mouri˜no, J., and Gerstner, W.
(2003). Non-invasive brain-actuated control of a mo-
bile robot. In Proceedings of the 18th International
Joint Conference on Artificial Intelligence, Acapulco,
M´exico.
Pfurtscheller, G. and da Silva, F. H. L. (1999). Event-
related eeg/meg synchronization and desynchroniza-
tion: basic principles. Clinical Neurophysiology,
110(11):1842–1857.
Rangaraj, M. R. (2002). Biomedical Signal Analysis. Wiley
Interscience, New York.
Rebsamen, B., Burdet, E., Guan, C., Zhang, H., Teo, C. L.,
Zeng, Q., Laugier, C., and Ang Jr., M. H. (2007). Con-
trolling a wheelchair indoors using thought. IEEE In-
telligent Systems, 22:18–24.
Tanaka, K., Matsunaga, K., and Wang, H. (2005).
Electroencephalogram-based control of an electric
wheelchair. IEEE Transactions on Robotics, 21:762–
766.
Webster, J. G. (1998). Medical Instrumentation. Applica-
tion and Design. John Wiley & Sons, Inc., New York,
3 edition.
Wolpaw, J. R., Birbaumer, N., McFarland, D. J.,
Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-
computer interfaces for communication and control.
Clin Neurophysiol, 113(6):767–791.
A VERSATILE ROBOTIC WHEELCHAIR COMMANDED BY BRAIN SIGNALS OR EYE BLINKS
67