Improving Isometric Hand Gesture Identification for

HCI based on Independent Component Analysis in

Bio-signal Processing

Ganesh R. Naik

1, 2

, Dinesh K. Kumar

, Hans Weghorn

, Vijay P. Singh

and Marimuthu Palaniswami

School of Electrical and Computer Engineering, Royal Melbourne Institute of Technology

University, GPO BOX 2476V, Melbourne, Victoria-3001, Australia

BA-University of Cooperative Education, PO BOX 10 05 63, 70004, Stuttgart, Germany

Department of Electrical and Electronic Engineering, The University of Melbourne

Parkville, Victoria-3010, Australia

Abstract. Hand gesture identification has various human computer interaction

(HCI) applications. There is an urgent need for establishing a simple yet robust

system that can be used to identify subtle complex hand actions and gestures

for control of prosthesis and other computer assisted devices. Here, an approach

is explained to demonstrate how hand gestures can be identified from isometric

muscular activity, where signal level is low and changes are very subtle. Obvi-

ous difficulties arise from a very poor signal to noise ratio in the recorded elec-

tromyograms (EMG). Independent component analysis (ICA) is applied to

separate these low-level muscle activities. The order and magnitude ambiguity

of ICA have been overcome by using a priori knowledge of the hand muscle

anatomy and a fixed un-mixing matrix. The classification is achieved using a

back-propagation neural network. Experimental results are shown, where the

system was able to reliably recognize motionless gestures. The system was

tested across users to investigate the impact of inter-subject variation. The ex-

perimental results demonstrate an overall accuracy of 96%, and the system was

shown being insensitive against electrode positions, since these successful ex-

periments were repeated on different days. The advantage of such a system is,

that it is easy to train by a lay user, and that it can easily be implemented as

real-time processing after an initial training. Hence, EMG-based input devices

can provide an effective solution for designing mobile interfaces that are subtle

and intimate, and there exist a range of applications for communication, emo-

tive machines and human computer interface.

1 Introduction

Hand gesture identification has numerous human computer interface (HCI) applica-

tions related to controlling machines and computers. Some of the commonly em-

ployed techniques include mechanical sensors [1], vision-based systems [2] and the

R. Naik G., K. Kumar D., Weghorn H., P. Singh V. and Palaniswami M. (2007).

Improving Isometric Hand Gesture Identiﬁcation for HCI based on Independent Component Analysis in Bio-signal Processing.

In Proceedings of the 7th International Workshop on Pattern Recognition in Information Systems, pages 171-180

DOI: 10.5220/0002427301710180

 SciTePress

use of electromyogram [3]. Electromyogram has an advantage of being easy to re-

cord, and is non-invasive. The Electromyogram is an electrical signal generated by

muscular contraction [4]. It is a result of the spatial and temporal integration of the

motor unit action potential (MUAP) originating from different motor units. It can be

recorded non-invasively using surface electrodes in different pairs, each pair consti-

tuting a channel.

In previous research, gestures are typically sensed by accelerometers [5], capaci-

tive techniques [6] or proximity sensors worn on different parts of the body [7].

These techniques require the users to noticeably move their limbs, which can be in-

convenient and socially unacceptable. On the contrary, electromyographic (EMG)

signals can convey information about isometric muscular activity: activity related to

very subtle or no movement at all. Hence it allows the definition of a class of “subtle”

or “motionless gestures” that can be used to design discreet, intimate mobile inter-

faces.

EMG is a biosignal related to muscle contraction. Studies on the use of EMG for

gesture recognition have been reported, but none of them takes explicit advantage of

its subtlety, the fact that commands can be issued without the generation of observ-

able movements.

Any hand movement is a result of a complex combination of many flexors and ex-

tensors present in the forearm. Since all these muscles present in the forearm are close

to each other, myo-electric activity observed from any muscle site comprises the

activity from the neighbouring muscles as well, referred to as cross-talk. When the

muscle activity is small (subtle), the signal strength is small and the impact of cross

talk and noise is very high. This is further exaggerated when considering different

subjects, since the size of the muscles, presence of subcutaneous fat layer and also the

training level is different for different people. Therefore this mixing of electrical

activity from different muscles to result in the surface EMG (sEMG) signal can not be

easily modelled or generalized. Extraction of the useful information from such kind

of surface EMG becomes even more difficult for low level of contraction mainly due

to the low signal – to – noise ratio. At low level of contraction, EMG activity is

hardly discernible from the background activity. Therefore to correctly classify the

movement and gesture of the hand more precisely, EMG needs to be decomposed to

identify activities of individual muscles. There is little or no prior information of the

muscle activity, and the signals have temporal and spectral overlap, making the prob-

lem suitable for blind source separation (BSS) or ICA for the separation of muscle

activities.

ICA is an iterative technique where the only model of the signals is the independ-

ence, and the distribution. The outcome of ICA is that the signals are separated with-

out there being any information about the order of the sources. While this difficulty is

generally not consequential for audio signals, this would be of concern when working

with muscle activity. The spatial location of the active muscle activity is the determin-

ing factor of the hand action and gesture. To overcome this difficulty, one approach

that has been reported is the use of prior knowledge of the muscle anatomy. The ad-

vantage of this approach is the model based approach that provides a well defined

muscle activity pattern.

In the current technology any mobile device should be as natural and conceptually

as un noticeable as possible. Hence our research extends this concept: we believe that

172

not only the devices should be unnoticeable and natural, but also the interaction with

them needs to be subtle and discreet. Therefore, we promote the idea of subtle ges-

tures (isometric hand gesture identification).

2 Hand Gesture Identification for HCI and Related Work

Computers and computerised machines have become a new element of our society.

Human-computer interaction requires the design, and implementation of interactive

computing systems for human use. The intent is to provide seamless and natural inter-

face that allows the human user to control and interact with computers and computer

based machines.

The use of hand gesture provides an attractive alternative to cumbersome interface

devices for human computer interaction applications. Human hand gestures are a

mean of non-verbal interaction among people. They range from simple actions of

pointing at objects to the more complex ones that express our feelings and communi-

cate with others. Numerous approaches have been applied to the problem of visual

interpretation of gestures for HCI. Many of those approaches have been chosen and

implemented to focus on a particular aspect of gestures: Hand tracking, pose classifi-

cation, or hand posture interpretations [2].

A number of approaches based on hand gesture identification have been proposed

for human computer interaction. Wheeler et al. demonstrated that neuroelectric joy

sticks and keyboards can be used for HCI [8]. Trejo et al [9] developed a technique

for multi modal neuroelctric interface. The most recent work includes the investiga-

tion of eleven normally limbed subjects (eight males and four females) for six distinct

limb motions: wrist flexion, wrist extension, supination, pronation, hand open, and

hand close. Each subject underwent four 60-seconds sessions, producing continuous

contractions [10]. Recent studies focus on the use of EMG for the recognition of an

alphabet of discrete gestures. Fistre and Tanaka [11] propose a system that can recog-

nize six different hand gestures using two EMG channels on the forearm. The device

is designed to control consumer electronics and is described as portable.

Wheeler and Jorgensen [8] report the development and successful testing of a neu-

roelectric joystick and a neuroelectric keypad. By using EMG signals collected from

four and eight channels on the forearm they successfully recognise the movement

corresponding to the use of a virtual joystick and virtual numeric keypad. Gestures

mimicking the use of physical devices are successfully recognised using hidden

Markov models.

To improve the reliability, a number of efficient solutions to gesture input in HCI

exist such as:

• Restrict the recognition situation.

• Use of input devices (e.g. data glove).

• Restrict the object information.

• Restrict the set of gestures.

In traditional HCI, most attempts have used some external mechanical device such

as an instrumented glove. If the goal is natural interaction in everyday situations this

might not be acceptable. Vision based approach to hand-centered HCI has been pro-

173

posed in recent years. However vision based techniques require restricted back-

grounds and camera positions and are suitable for a small set of gestures performed

with only one hand [1]. In this report we propose the identification of maintained

hand gesture based on the muscle activity using the decomposition of surface EMG. It

is a combination of model based approach with blind source separation

3 Foundation of Semg Bio-signal Processing

Surface EMG (sEMG) is a result of the superposition of a large number of transients

(muscle action potentials) that have temporal and spatial separation that is pseudo-

random. The origin of each of the MUAP is inherently random and the electrical

characteristics of the surrounding tissues are non-linear. Due to the nature of this

signal the amplitude of the EMG signal is pseudo-random and the shape of the prob-

ability distribution function resembles a Gaussian function.

sEMG is a non-invasive recording, requires relatively simple equipment, and this

opens it for numerous applications. The close relationship of surface EMG with the

force of contraction of the muscle is useful for number of applications such as sports

training and for machine control. The relationship of surface EMG spectrum with

muscle fatigue is also very useful for occupational health and sports training.

One property of the surface EMG is that the signal originating from one muscle

can generally be considered to be independent of other bioelectric signals such as

electrocardiogram (ECG), electro-oculargram (EOG), and signals from neighbouring

muscles. This opens an opportunity of the use of independent component analysis

(ICA) for this application.

3.1 Independent Component Analysis

Independent component analysis one of the Blind source separation (BSS) technique,

aims at recovering the sources from a set of observations. Applications include sepa-

rating individual voices in cocktail party. In BSS problem, it contains two processes.

They are the mixing process and un-mixing process. First, we observe a set of multi-

variate signals x = [x

(t), x

(t),…,x

(t)]

that are assumed to be linearly mixed with a

set of source signals s = [s

(t), s

(t),…, s

(t)] The mixing process is hidden so we can

only observe the mixed signals. The task is to recover the original source signals from

the observations through a un-mixing process. Equation 1 and 2 describe the mixing

and un-mixing processes mathematically.

Mixing x = As (1)

Un-mixing Wx = WAs (2)

For solving the BSS it is assumed that the number of observations is equal to the

number of source signals. Matrix s contains the original source signals driving the

observations, whereas the separated signals are stored in matrix u. They are both

[n×t] matrices. A and W are both [n×n] matrices, called mixing and un-mixing matrix

174

respectively. If the separated signals are the same as the original sources, the mixing

matrix is the inverse of the un-mixing matrix, i.e. A = W

-1

ICA is an iterative method that is able to separate independent sources from the

mixture [12]. ICA estimates the mixing matrix W using ‘independence’ based cost

function. Various ICA algorithms have been proposed. Most of them use higher order

statistics to obtain the independent components [12].

3.2 Relevance of ICA for Surface EMG Signal Evaluation

The goal of this section is to demonstrate that there is a strong theoretical basis for

applying ICA to sEMG. The assumptions that underpin the theory of instantaneous

ICA, indicate that ICA is ideally suited to separating sources when

• The sources are statistically independent

• Independent components have non-Gaussian distribution

• The mixing matrix is invertible.

These assumptions are well satisfied by sEMG data as MUAPs are statistically in-

dependent, have non-Gaussian distributions and we can be (virtually) certain that the

mixing matrix will be invertible. There are, however, two other practical issues that

must be considered. Firstly, to ensure that the mixing matrix is constant the sources

must be fixed in space (this is an implied assumption as only the case of a constant

mixing matrix is considered). This is satisfied by sEMG as motor units are in fixed

physical locations within a muscle, and in this sense applying ICA to sEMG is much

simpler than in other biomedical signal processing applications such as EEG or fMRI

in which the sources can move [13]. Secondly, in order to use ICA it is essential to

assume that signal propagation time is negligible. Volume conduction in tissue is

essentially instantaneous [14]. Hence this assumption is also well satisfied.

Based on the above discussion of the ICA assumptions as they apply to sEMG, it is

reasonable to be confident that ICA can be effectively applied to EMG data. The

validity of using ICA on sEMG is examined later in the experimental and analysis

section.

4 Methodology

4.1 Experimental Procedure

University ethics committee granted approval to conduct experiments on human sub-

jects and acquire Surface EMG using surface electrodes. For the hand gesture ex-

periments four subjects whose ages ranging from 21 to 32 years (three males and one

female) were chosen. For the data acquisition a proprietary Surface EMG acquisition

system by Delsys (Boston, MA, USA) was used. Four electrode channels were placed

over four different muscles as indicated in the Table 1 and Fig. 1. A reference elec-

trode was placed at Epicondylus Medialis.

175

Table 1: Muscle Electrode Configuration.

Channel Muscle Function

1 Brachioradialis Flexion of forearm

2 Flexor Carpi radialis (FCR) Abduction and flexion of wrist

3 Flexor Carpi Ulnaris (FCU) Adduction and flexion of wrist

4 Flexor digitorum superficialis (FDS) Finger flexion while avoiding wrist flexion

Fig. 2. Hand gesture experimental set up with four electrodes.

Each channel is a set of two differential electrodes with a fixed inter-electrode dis-

tance of 10mm and a gain of 1000. Before placing the electrodes subject's skin was

prepared by lightly abrading with skin exfoliate to remove dead skin that helps in

reducing the skin impedance to less than 60 kilo Ohm. Skin was also cleaned with

70% v/v alcohol swab to remove any oil or dust on the skin surface.

ICA is suitable when the numbers of recordings are same as or greater than the num-

ber of sources. This paper reports using 4 channels of EMG recorded during hand

actions that required not greater than 4 independent muscles. This ensures that the un-

mixing matrix is a square matrix of size of 4×4. The experiments were repeated on

two different days. Subjects were asked to keep the forearm resting on the table with

elbow at an angle of 90 degree in a comfortable position. Four isometric hand actions

were performed and repeated 12 to 14 times at each instance. Each time raw signal

sampled at 1024 samples/second was recorded. Markers were used to obtain the Iso-

metric contraction signals during recording. A suitable resting time was given be-

tween each experiment. There was no external load. The actions were complex to

determine the ability of the system when similar muscles were active simultaneously.

The four different hand actions were performed and are listed below:

• Middle and index finger flexion.

• Little and ring finger flexion

• All finger flexion

• Finger & wrist flexion together.

These hand actions were selected based on small variations between the muscle ac-

tivities of the different digitas muscles situated in the forearm.

4.2 Data Analysis

The aim of this experiment was to test the use of ICA along with known properties of

the muscles for separation of sEMG signals for the purpose of identifying stationary

hand gestures and finger movement actions. Each action was repeated 12 to 14 times

176

and each contraction lasted approximately 2.5 seconds. The sampling rate was 1024

samples per second, and this gives approximately 2500 samples during the contrac-

tion. There were four channel (recordings) electrodes over the four active muscles

associated with the different hand gestures, forming a square 4×4 mixing matrix. The

sEMG recordings were then separated using fast ICA algorithm which is developed

by the team at the Helsinki University of Technology [15]. The mixing matrix A was

computed for the first set of data only and kept constant throughout the experiment.

The independent sources of motor unit action potentials that mix to make the EMG

recordings were computed using the following equation:

s = Bx (3)

where B is the inverse of the mixing matrix A. This process was repeated for each of

the hand gestures. Four sources were estimated for each experiment. The example of

four channel source separation using Fast ICA matlab package is depicted in Fig. 3.

Fig. 3. Estimated four channel source signals s(t) from a four channel recording x(t)-1024

sampling rate using fast ICA.

After separating the four sources sa, sb, sc and sd, each of these was segmented to

2500 samples length. Root Mean Squares (RMS) was computed for each separated

sources using the following relation:

Srms

∑

(4)

where s is the source and N is the number of samples (N = 2500). This results in one

number representing the muscle activity for each channel for each hand action.

RMS value of muscle activity of each source represents the muscle activity of that

muscle and is indicative of the force of contraction generated by each muscle. Taking

a ratio of these activities gives a relative combination of the activity from each of

these muscles and has been used to identify the hand gesture. A constant mixing ma-

trix A and set of weight matrix for neural networks were used for each subject making

the system configured for each individual.

177

The above process was repeated for all four different hand actions 12 to 14 times

and for each of the participants. These 12 to 14 sets of examples were used to train a

back-propagation neural network with 4 inputs and 3 outputs. The 4 RMS (Root

Mean Square) values of the muscles were the input and the 3 RMS (Root Mean

Square) values were the output. In the first part of the experiment, RMS values of

recordings for each subject were used to train the ANN classifier with back-

propagation learning algorithm. The second part of the experiment (testing) was to

verify the training results. For that the set of data’s which were not used for the train-

ing purpose (an independent data set) was selected. During the training, ANN con-

sisted of two hidden layers with a total of 20 nodes and a sigmoid function as thresh-

old function. The gradient descent algorithm with a learning rate of 0.05 was used to

avoid any chances of local minima. During testing, the ANN with weight matrix

generated during training, was used to classify RMS of the muscle activity separated

using un-mixing matrix generated during training. The ability of the network to cor-

rectly classify the inputs against known hand actions were used to determine the effi-

cacy of the technique.

4.3 Results and Observations

The results of the experiment demonstrate the performance of the above described

system. The results of testing the back propagation ANN to correctly classify the test

data based on the weight matrix generated using the training data is tabulated in Table

2. The accuracy was computed based on the percentage of correct classified data

points to the total number of data points. These results indicate an over all classifica-

tion accuracy of 96% for all the experiments. The results demonstrate that this tech-

nique can be used for the classification of different types of isometric muscular activ-

ity. This feature makes it possible to define a class of subtle motionless gestures to

control an interface without being noticed and without disrupting the surrounding

environment.

Table 2. Experimental results for Isometric Hand Gesture Identification.

Number of

participants

Middle and

index finger

flexion

Little and ring

finger flexion

All finger

flexion

Finger and

wrist flexion

together

Subject 1 97% 96% 97% 96%

Subject 2 96% 96% 96% 96%

Subject 3 97% 96% 96% 96%

Subject 4 97% 97% 96% 97%

5 Discussion

The proposed technique is capable of classifying small levels of muscle activity to

identify Isometric hand gesture. Its base is using a combination of independent com-

178

ponent analysis (ICA), known muscle anatomy and neural network configured for the

individual. The results indicate the ability of the system to perfectly recognize the

hand gesture even though the muscle activity is very low and there are number of

active muscles for each of the gestures.

There exist numerous papers in literature, which have attempted to identify hand

and body gestures from sEMG recordings, but all come with low reliability, perhaps

due to low signal to noise ratio and large cross-talk between different simultaneously

active muscles. In the recent past, ICA has been applied to separate the muscle activ-

ity and to reduce noise to overcome this difficulty, but the order and magnitude ambi-

guity makes the technique unreliable.

This research overcomes these issues by using a priori knowledge of the anatomy

of muscles in combination with blind source separation technique. Using a combina-

tion of the model, and ICA approaches with a neural network configured for the indi-

vidual overcomes the order and magnitude ambiguity.

6 Conclusions and Future Work

This investigation has shown that a combination of a known biological model used in

a semi-blind ICA combined with neural networks for classification can effectively be

employed to detect small muscle activities, and by that to identify subtle hand actions

and gestures. The presented experimental methods are able to reliably recognize a

motionless gesture for different muscle volumes.

A new approach that combines semi-blind ICA and a back-propagation neural

network was used to separate and identify subtle hand gestures, and subsequently

using the combination of the mixing matrix and network weights to classify the

sEMG recordings in almost real-time.

The results demonstrate that the technique can be effectively used to identify hand

gestures based on surface EMG when the level of activity is very small. The gestures

have been chosen, because each of these represents a complex combination of muscle

activations and can be extrapolated for a larger number of gestures. Nevertheless, it is

important to test the technique for more actions and gestures, and for a large group of

people. In parallel, there is ongoing work to investigate recognition of gestures on a

larger number of people and for a greater variety of hand actions to increase the per-

formance of the system.

We are working on expanding the EMG gesture for extended levels of control.

While further work on the signal processing may make it possible to recognize multi-

ple subtle gestures from a single muscle, it appears more practical to define a more

extended interface using different controllers on various muscles (e.g. on both arms).

Future work also shall include conducting experiments on inter-day and intra-day

variations to verify the stability of the system and also to develop a portable model

for hand gesture recognition using semi-blind ICA technique.

Overall, the purpose of this project is to develop new perceptual interfaces for

human computer interaction based on hand gesture identification, and to investigate

how such interfaces can complement or replace traditional interfaces based on key-

boards, mice, remote controls, data gloves and speech. Application fields for hand

gestures analysis include control of consumer electronics, interaction with visualiza-

tion systems, control of mechanical systems, and computer games.

179

One important benefit of such an HCI approach is that visual information makes it

possible to communicate with computerized equipment at a distance, without a need

for physical contact to the controlled target. Compared to speech commands, hand

gestures are especially advantageous in noisy environments –particularly in situations

where speech commands would be disturbed – as well as for communicating quantita-

tive information and spatial relationships. Furthermore, the human user shall be en-

abled to control electronic systems in a quite natural manner, without requiring spe-

cialized external equipment.

References

1. Pavlovic, V. I., Sharma, R., Huang, T. S.: Visual interpretation of hand gestures for human

computer interaction, Vol. 19. IEEE Transactions on Pattern Analysis and Machine Intelli-

gence (1997) 677 – 695

2. Rehg, J. M., Kanade, T.: Vision-based hand tracking for human-computer interaction,

IEEE, Workshop on Motion of Non-Rigid and Articulated Objects (1994) 16 - 22

3. Koike, Y., Kawato, M.: Human Interface Using Surface Electromyography Signals, Vol.

79. Electronics and Communications, Japan (1996) 15 – 22

4. Luca, C. J. D.: Physiology and mathematics of myoelectric signals. Vol. 266. IEEE Trans-

actions on Biomedical Engineering (1979)

5. Oakley, I., Angesleva, J., Hughes, S., O'Modhrain, S.: Tilt and Feel: Scrolling with Vibro-

tactile Display, The EuroHaptics (2004)

6. Rekimoto, J.: Gesture Wrist and Gesture Pad: Unobtrusive Wearable Interaction Devices,

Fifth IEEE International Symposium on Wearable Computers (2001)

7. Headon, R., Coulouris, G.: Supporting Gestural Input for Users on the Move, IEE Eu-

rowearable (2003) 107-112

8. Wheeler, K. R., Jorgensen, C. C.: Gestures as input: Neuroelectric joysticks and keyboards.

Vol. 2. IEEE Pervasive Computing (2003) 56–61

9. Trejo, L. J., Wheeler, K. R., Jorgensen, C. C., Rosipal, R., Clanton, T. S., Matthews, B.,

Hibbs, A. D., Matthews, R., Krupka, M.: Multimodal neuroelectric interface development,

Vol. 11. IEEE Transactions on Neural Systems and Rehabilitation Eng. (2003) 199– 204

10. Chan, A. D. C., Englehart, K. B.: Continuous myoelectric control for powered prostheses

using hidden markov models, Vol. 52. IEEE Transactions on Biomedical Engineering

(2005) 121–124

11. Fistre, J., Tanaka, A.: Real time EMG gesture recognition for consumer electronics device

control, Presented at Sony CSL Paris Open House, http://www.csl.sony.fr/ atau/gesture/

(2002)

12. Hyvarinen, A., Karhunen, J., Oja, E.: Independent Component Analysis, John Wiley, New

York (2001)

13. Jung, T., Makeig, S., Lee, T., McKeon, M., Brown, G., Bell, A., Sejnowski, T. : Independ-

ent component analysis of biomedical signals, Second International Workshop on Inde-

pendent Component Analysis and Blind Signal Separation (2001) 633–644

14. Makeig, S., Jung, T., Bell, A., Sejnowski, T.: Independent component analysis of electro-

encephalographic data, Vol. 8. Advances in neural information processing systems (1996)

145–151

15. Hyvarinen, A., Oja, E.: Fast ﬁxedpoint algorithm for independent component analysis, The

Fast ICA matlab software package, http://www.cis.hut.ﬁ/projects/ica/fastica/index.shtml

(1997)

180