Human Activity Recognition for an Intelligent Knee Orthosis

Diliana Rebelo

, Christoph Amma

, Hugo Gamboa

1,3

and Tanja Schultz

CEFITEC, Physics Department, FCT-UNL, 2829-516, Caparica, Portugal

Cognitive System Lab (CSL), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

PLUX-Wireless Biosignals S.A, 1050-059, Lisbon, Portugal

Keywords:

Biosignals, Human Activity Recognition, Signal-processing, Hidden Markov Models.

Abstract:

This paper investigates the possibility to classify isolated human activities from biosignal sensors integrated

into a knee orthosis. An intelligent orthosis that is capable to recognize its wearers activity would be able to

adapt itself to the users situation for enhanced comfort. We use a setup with three modalities: accelerometry,

electromyography and goniometry to measure leg motion and muscle activity of the wearer. We segment

signals in motion primitives and apply Hidden Markov Models to classify these isolated motion primitives.

We discriminate between seven activities like for example walking stairs and ascend or descend a hill. In a

small user study we reach an average person-dependent accuracy of 98% and a person-independent accuracy

of 79%.

1 INTRODUCTION

Passive orthoses are widely used for conservative

therapy of diseases like Gonarthrosis, which is Os-

teoarthritis in the knee joint. Depending on the type

of arthrosis, these orthoses apply a constant force on

the knee joint to correct a defective position. This is

usually painful or at least unpleasant for the wearer

after a certain amount of time. Future active orthoses

could be able to vary this force depending on the cur-

rent wearers activity and therefore only apply force

if the knee joint is stressed, e.g. while walking stairs

but not while the wearer is sitting. This will impose

less strain on the wearer. In order to develop such a

system, it will be necessary to recognize the wearers

activity, which is the topic of this paper. The study

presented in this paper is part of a larger effort to de-

velop such active orthoses.

In this paper we evaluate the possibility to recog-

nize human activities based on data from biosignal

sensors solely placed on or under an existing passive

knee orthosis to allow future integration of sensors.

The contribution of this paper is the evaluation of the

ability to recognize activities with these restrictions

on sensor placement as well as providing a proof-of-

concept for the development of an activity recognition

system for an intelligent orthosis.

The focus of this work is the question how well we

can discriminate already segmented isolated motion

primitives. In case of periodic activities like walk-

ing, we deﬁne a primitive as one cycle, in case of

non-periodic activities like sitting down, we deﬁne a

primitive as the complete motion. Therefore, prior

to classiﬁcation, the continuous data recordings need

to be segmented into isolated motion primitives. We

use an automatic approach for the segmentation of all

periodic activities and perform manual segmentation

for non-periodic activities. We use Hidden-Markov-

Models (HMM) as classiﬁer and model each of the

seven motion primitives with one HMM. In future

work we want to rely on the implicit segmentation

ability of HMMs to allow for realtime usage.

2 RELATED WORK

Gait and posture are often categorized as the standard

human movement from which analysis allow clinical

evaluation. In the past, some studies have proved the

reliability of accelerometry on activity recognition on

data collected from different areas of the body. For

example, in (Mathie et al., 2003), the use of triax-

ial accelerometers successfully distinguished between

activity (transitions sit-to-stand and stand-to-sit and

walk) and rest. In the context of physical activity

recognition, in (Welk and Differding, 2000), seden-

tary activities such as sitting or sleeping are discrimi-

nated from moderate intensity activities such as walk-

368

Rebelo D., Amma C., Gamboa H. and Schultz T..

Human Activity Recognition for an Intelligent Knee Orthosis.

DOI: 10.5220/0004254903680371

In Proceedings of the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS-2013), pages 368-371

ISBN: 978-989-8565-36-5

 2013 SCITEPRESS (Science and Technology Publications, Lda.)

ing through the analysis of acceleration data.

The use of EMG or electrogoniometers have been ap-

plied mostly on kinematics evaluation (Rowe et al.,

2000) or on pattern comparisons for diagnosis (Suda,

2011).

However, these aforementioned researches use only

accelerometers, goniometers or EMG sensors sepa-

rately with less restrictions on sensor placement.

Hidden Markov Models (HMM) are widely used

for activity recognition (Lukowicz et al., 2004). They

are especially useful if motions can be modeled as a

sequence of motion primitives like for example ges-

tures from a gesture alphabet (Amma et al., 2012).

3 DATA ACQUISITION AND

SIGNAL PROCESSING

3.1 Experimental Setup

We equipped a standard Ortema

ipomax passive or-

thosis with two triaxial accelerometers and one bi-

axial goniometer. Additionally, six EMG sensors to

measure muscle activity were put on the skin under

the orthosis. It is a requirement of our application

scenario to place all sensors on or under the ortho-

sis, since all sensors should be integrateable into the

orthosis in the future. The orthosis itself consists of

two rigid shells, one for the upper and one for the

lower leg which are ﬁxed to the leg with straps. The

shells are connected by a joint on each side in order

to allow ﬂexion of the knee joint. The accelerometers

were placed on the frontside of the orthosis, one on

the lower part and one on the upper part to measure

acceleration of tibia and femur respectively. The go-

niometer was placed on the joint so it can measure the

angle of the orthosis joint which can be assumed equal

to the angle of the knee joint. Although the goniome-

ter is biaxial only the axis aligned to the rotation axis

of the orthosis joint was used. EMG electrodes were

placed on the knee’s main extensor and ﬂexor mus-

cles: semimembranosus, semitendinosus, gastrocne-

mius (external and internal), vastus lateralis and vas-

tus medialis. Figure 1 shows the EMG electrodes’

locations (A, B) as well as the complete experimental

setup (C).

The signals were recorded wirelessly with two

synchronized bioPluxresearch

devices at a sampling

rate of 1000Hz.

We performed a study with 6 healthy male subjects in

the age between 20 and 30. Although the orthosis was

www.ortema.de

www.plux.info

Figure 1: (A, B) Placement of electromyography surface

electrodes (C) Complete experimental setup: two Bioplux

research devices, two ACC (circles), six EMG and one go-

niometry sensors (arrow) placed on a regular Ortema right

knee orthosis.

not adapted to the anatomy of the subjects, all partic-

ipants were able to move their knee freely and didn’t

feel any pain. Additionally, we made video record-

ings of the experiments for reference.

3.2 Acquisition Protocol

The subjects performed the following 7 activities for

the given number of repetitions:

• Activity 1. Sit and stand (20 repetitions);

• Activity 2. Sit, stand and walk a few steps (5 rep-

etitions);

• Activity 3. Walk to a chair, turn, sit and stand (20

repetitions);

• Activity 4. Walk and stairs up (10 repetitions);

• Activity 5. Walk and stairs down (10 repetitions);

• Activity 6. Ascend a hill (2 repetitions of 2 min-

utes each, with 27% of inclination);

• Activity 7. Descend a hill (2 repetitions of 2 min-

utes each, with 27% of inclination);

Activities 6 and 7 were performed on a treadmill with

an adjustable inclination at a walking speed of 2.5

km/h.

The combined activities were chosen to have more

challenging data for future evaluation of continuous

recognition of activities. Since this is not in the

scope of this paper the recorded data was segmented

into motion primitives. In total, we work with 7

movements: walk (W), stand-to-sit (St), sit-to-stand

(Si), stairs up (Su), stairs down (Sd), ascend (A) and

descend (D).

3.3 Data Segmentation

The purpose of the data segmentation step is to split

the continuous sensor recordings into motion primi-

HumanActivityRecognitionforanIntelligentKneeOrthosis

369

tives. For the periodic motions walk, stairs up, stairs

down, and ascend and descend a hill, we deﬁne a mo-

tion primitive as one complete gait cycle. For motions

like stand-to-sit and sit-to-stand, we deﬁne a motion

primitive as the movement that occurs between the

two static postures “stand” and “sit”. We segment

each continuous periodic signals in its unique mo-

tion primitives. We segment the periodic activities ac-

cording to the standard segmentation of gait cycles in

(Sutherland, 2002). A gait cycle consists of a stance

and a swing phase and starts and ends with the initial

contact of the heel on the ﬂoor. At this point of the

gait cycle the knee angle is minimal.

For cyclic motions we compute the minima of the

goniometer signal and take these as segment borders.

Figure 2 illustrates the resulting segmentation. The

Figure 2: Example segmentation of the activity walk shown

for the goniometer signal. The minima represent segment

borders.

non-periodic movements stand-to-sit and sit-to-stand

were segmented manually based on the goniometer

signal. We visually checked the results of the auto-

matic segmentation to verify the correctness of our

data segmentation.

3.3.1 Data Corpus

After acquisition and segmentation of all recorded

data, the segments for the seven different activities

form our data corpus. In order to get a balanced

dataset for the evaluation we randomly chose 17 seg-

ments per person for each activity, which is the min-

imum number of samples we recorded over all ac-

tivities. The resulting balanced data corpus therefore

consists of 119 segments per person and 714 segments

in total.

3.4 Classiﬁcation and Feature

Extraction

For feature extraction, the signal is windowed using

a rectangular window function with 50 ms window

length and no window overlap. All features are com-

puted on the resulting windows. We denote the sam-

ples in one window by (x

, . . . , x

) where N is the

total number of samples per window. Along the ex-

periments, for each window, we extract the Average

avg =

∑

N−1

k=0

for the goniometer and accelerome-

ter’s output and the Root Mean Square (RMS) RMS =

+ x

+ . . . + x

) for the EMG signals.

We use continuous density Hidden Markov Mod-

els (HMM) as classiﬁer (Rabiner, 1989). Each mo-

tion primitive is modeled by one HMM with a left-to-

right topology with 10 states and Gaussian-Mixture-

Models with two components. In future work we

want to evaluate the ability to concatenate the primi-

tive models to recognize continuous activities without

the need for explicit segmentation. We use the BioKit

toolbox developed at the Cognitive Systems Lab for

the HMM training and decoding.

4 EXPERIMENTS

We evaluate the classiﬁcation accuracy for classi-

fying isolated motion primitives for all modality

combinations in a subject-dependent and a subject-

independent context.

4.1 Experiment 1: Subject-dependent

To evaluate the subject-dependent accuracy of the

classiﬁer we performed a leave-one-out cross-

validation for each subjects data. We reach accura-

cies between 65% and 98% depending on the signal

modalities used. Table 1 shows the results from which

we can state that the combination GON&ACC has the

highest accuracy.

Table 3 shows the confusion matrix for the best

performing signal combination GON&ACC.

Table 2 shows the classiﬁcation accuracy per sub-

ject for the best set. High accuracies are reached for

all subjects. For the subject-dependent case, we be-

lieve that a robust activity recognition is possible.

Table 1: Classiﬁcation accuracy (mean and standard devia-

tion) per combination of signals in subject’s dependent and

independent context.

Classiﬁcation Accuracy (%)

Subject dependent Subject independent

EMG 65.41 ± 0.17 44.57 ± 13.87

GON 83.33 ± 0.12 61.46 ± 11.67

ACC 97.34 ± 0.01 75.05 ± 22.65

EMG&GON 72.13 ± 0.17 47.94 ± 14.60

GON&ACC 98.32 ± 0.05 78.70 ± 21.64

EMG&ACC 81.23 ± 0.14 55.52 ± 14.46

All 87.54 ± 0.13 63.66 ± 17.14

BIOSIGNALS2013-InternationalConferenceonBio-inspiredSystemsandSignalProcessing

370

Table 2: Classiﬁcation accuracy (mean and standard devi-

ation) per subject for the GON&ACC set in a subject’s de-

pendent and independent context.

Classiﬁcation Accuracy (%)

Subjects Subject dependent Subject independent

1 97.32 ± 0.04 36.13 ± 21.43

2 99.11 ± 0.02 83.19 ± 10.11

3 97.32 ± 0.04 89.08 ± 4.56

4 98.22 ± 0.03 78.15 ± 8.17

5 98.22 ± 0.03 94.12 ± 0.15

6 99.11 ± 0.04 91.52 ± 0.87

Table 3: Confusion matrix for GON&ACC in a subject-

dependent and subject-independent (between brackets) con-

text, in percentage.

W St Si Su Sd A D

W 95(89) 0 0 0 3(4) 1(0) 1(7)

St 0 100(83) 0 0 0(17) 0 0

Si 0 1(0) 99(83) 0(17) 0 0 0

Su 0 0 0 98(92) 2(4) 0(4) 0

Sd 0(4) 0(1) 0(1) 0(9) 98(66) 0(1) 2(18)

A 0(1) 0 0 1(32) 0 99(67) 0

D 0(19) 0 0 0 1(9) 0 99(72)

4.2 Experiment 2: Subject-independent

We evaluate the subject-independent performance

with a leave-one-out cross-validation on the subjects.

That means we test on one subjects data and train

on the data of the remaining subjects. We use all

samples in our database, resulting in 595 samples for

training and 119 samples for testing for each subject.

We reach accuracies between 49% and 79% for

the different signal modalities. Table 1 shows the

results and, analogously to the subject-dependent

evaluation, we can state that the combination

GON&ACC has the highest accuracy. Nevertheless,

compared to the subject-dependent case, the accuracy

is much lower which can be explained by the varia-

tions in human motion for different subjects.

Table 3 shows the confusion matrix for

GON&ACC. We can see that the movements

stairs down and ascend are easily confused with

descend and stairs up, respectively. This is not

surprising since the performed motions are very

similar for these pairs of activities.

Table 2 shows the classiﬁcation accuracy per subject

for the GON&ACC set.

Concerning the evaluation per subject, the rec-

ognizer performed with an accuracy between 36%

and 94% with an average of 79%. Due to the small

number of subjects, the generalization ability of the

classiﬁer is relatively low and thus we can see a low

performance for subject 1.

5 CONCLUSIONS

In this work we evaluated the possibility to recog-

nize human activities from different biosignal sensors.

We reach a person-dependent accuracy of 98% and

a person-independent accuracy of 79%. The combi-

nation of GON&ACC signals gives the highest accu-

racy. Based on the dataset and the models for the mo-

tion primitives acquired in this work, we will investi-

gate continuous recognition of activities in the future,

which will be the next step towards an active intelli-

gent knee orthosis.

ACKNOWLEDGEMENTS

We would like to thank the KIT Sports Institute for

their support, Plux for supplying biosignal sensors

and the volunteers for the participation.

REFERENCES

Amma, C., Georgi, M., and Schultz, T. (2012). Airwriting:

Hands-free mobile text input by spotting and continu-

ous recognition of 3d-space handwriting with inertial

sensors. In Wearable Computers (ISWC), 2012 16th

International Symposium on, pages 52–59. IEEE.

Lukowicz, P., Ward, J., Junker, H., Stger, M., Trster, G.,

Atrash, A., and Starner, T. (2004). Recognizing work-

shop activity using body worn microphones and ac-

celerometers. In Pervasive Computing, volume 3001

of Lecture Notes in Computer Science, pages 18–32.

Springer.

Mathie, M., Coster, A., Lovell, N., and Celler, B. (2003).

Detection of daily physical activities using a triaxial

accelerometer. Medical and Biological Engineering

and Computing, 41(3):296–301.

Rabiner, L. (1989). A tutorial on hidden markov models

and selected applications in speech recognition. Pro-

ceedings of the IEEE, 77(2):257–286.

Rowe, P., Myles, C., Walker, C., and Nutton, R. (2000).

Knee joint kinematics in gait and other functional

activities measured using ﬂexible electrogoniometry:

how much knee motion is sufﬁcient for normal daily

life? Gait & posture, 12(2):143–155.

Suda, E. (2011). An

alise eletromiogr

aﬁca comparativa

de tornozelo durante a aterrissagem em jogadores de

olei com instabilidade cr

onica.

Sutherland, D. (2002). The evolution of clinical gait anal-

ysis: Part ii kinematics. Gait & posture, 16(2):159–

179.

Welk, G. and Differding, J. (2000). The utility of the digi-

walker step counter ro assess daily physical activity

patterns. Medicine and Science in Sports and Exer-

cise, 9(32):481–488.

HumanActivityRecognitionforanIntelligentKneeOrthosis

371