Sensor-based Pattern Recognition Identifying

Complex Upper Extremity Skills

Ryanne Lemmens

1,2

, Yvonne Janssen-Potten

1,2

, Annick Timmermans

1,3

Rob Smeets

1,2

and Henk Seelen

1,2

Department of Rehabilitation Medicine, Research School CAPHRI, Maastricht University, Maastricht, Netherlands

Adelante, Centre of Expertise in Rehabilitation and Audiology, Hoensbroek, Netherlands

BIOMED Biomedical Research Institute, Hasselt University, Hasselt, Belgium

1 OBJECTIVES

Objectively quantifying actual arm-hand

performance is very important to evaluate arm-hand

therapy efficacy in patients with neurological

disorders. Currently, objective assessments are

limited to evaluation of ‘general arm hand activity’,

whereas monitoring specific arm-hand skills is not

available yet. Instruments to identify skills and

determine both amount and quality of actual arm-

hand use in daily life are lacking, necessitating the

development of a new measure. To identify skills,

pattern recognition techniques can be used.

Commonly used pattern recognition approaches are:

statistical classification, neural networks, structural

matching and template matching (Jain et al., 2000).

The latter is used in the present study, aiming to

provide proof-of-principle of identifying skills,

illustrate this for the skill drinking in a standardized

setting and daily life situation in a healthy subject.

2 METHODS

Four sensor devices, each containing a tri-axial

accelerometer, tri-axial gyroscope and tri-axial

magnetometer were attached to the dominant hand,

wrist, upper arm and chest of participants. Thirty

healthy individuals performed the skill drinking 5

times in a standardized manner, i.e. with similar

starting position and instruction about how to

perform the skill. In addition, for one person a 30

minute registration in daily life including multiple

skills (of which 4 times the skill drinking) was

made.

Signals were filtered with a 4

order zero-time

lag low-pass Butterworth filter (cut off frequency:

2.5 Hz). Data analysis consisted of the following

steps: 1) temporal delimitation of each of the five

attempts of the skill drinking, i.e. identifying the

start and endpoint of each attempt recorded; 2)

normalization of the signals in the time domain in

order to correct for (small) variations due to

differences in speed of task execution; 3) averaging

signal matrices from the five attempts of each

individual person to obtain the individual template,

i.e. the underlying ensemble averaged signal matrix

per task per individual; averaging signal matrices

from the individual templates of multiple persons, to

create a generic template; 4) identification of

dominant sub phases of templates, within a specific

task, using Gaussian-based linear envelope

decomposition procedures; 5) recognition of specific

skill execution among various skills performed

daily, i.e. searching for template occurrence among

signal recordings gathered in a standardized setting

and a daily life condition, using feature extraction

and pattern recognition algorithms based on 2D

convolution. Cross-correlation coefficients were

calculated to quantify goodness-of-fit.

3 RESULTS

Performance of the skill drinking was identified

unambiguously (100%) in de standardized setting

(figure 1a). For the templates consisting of the

complete skill, mean cross-correlation was 0.93 for

the individual template and 0.79 for the generic

template. For the templates consisting of sub-phases,

mean cross-correlations ranged between 0.89 and

0.99 for the individual template and between 0.78

and 0.86 for the generic template.

In the daily life registration, all instances at

which drinking was performed, were recognized

with the template consisting of the complete skills

(mean cross-correlation: 0.51) (figure 1b). However,

also five false-positive findings were present (mean

cross-correlation: 0.46). Using the template

consisting of the sub phases, in general the skill

Lemmens R., Janssen-Potten Y., Timmermans A., Smeets R. and Seelen H..

Sensor-based Pattern Recognition Identifying Complex Upper Extremity Skills.

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

Individual

template

Generic

template

Combination of subtasks in time representing the task drinking:

Recognition of subphases of drinking: subphase 1; subphase 2; subphase 3; subphase 4

Recognition of complete skill drinking:

Drinking Eating Writing Call someone Put on one’s shoes Thumb through a book Vacuum cleaning

500 1000 1500 2000 2500 3000 3500

* Drink 3 times, without reaching ** Drinking: empty one’s glass in one drain

2000

4000 6000 8000 10000

2000 4000 6000 8000 10000

Identification of the skill drinking amongst

3 different skills in a standardized setting

***

Identification of the skill drinking

in daily life

Figure 1: Identification of the skill drinking in a daily activity registration in a standardized setting (A) and daily life setting

(B). Panel I displays the superimposed signals (36 in total) of a registration of an individual during the execution of several

skills. Panel B displays the pattern recognition using a individual and a generic template for the skill drinking. Both pattern

recognition with the complete skill as template and skill sub phases as template are shown. The black lines (complete skill)

and coloured lines (skill sub phases) in panel II mark the places were the template is recognised in the longer registration.

drinking was identified (cross-correlation ranging

between 0.62 and 0.82), but some sub phases were

not recognized correctly. A false-positive finding

occurred frequently for sub phase 1 and sporadic for

the other sub phases (mean cross-correlation

between 0.55 and 0.92). Regarding the combination

of sub phases, no false-positive findings were found.

4 DISCUSSION

Using this method, it is possible to identify a

specific skill amongst multiple skills, both in a

standardized setting and in a daily life registration.

The long-term aim is to use this method to a)

identify which arm-hand skills are performed during

daily life by individuals, b) determine the quantity of

skill execution, i.e. amount of use, and c) determine

the quality of arm-hand skill performance. At the

moment, as far as we know, no such instrument is

available. There are however many instrument being

developed using many different pattern recognition

techniques. Leutheuser et al, for example, used a

feature set of four time domain features and two

frequency domain features and a combination of

classification systems to distinguish between

activities like vacuuming, sweeping, sitting,

standing, bicycling, ascending/descending stairs and

walking (Leutheuser et al., 2013). Future research

will firstly focus on optimizing the method described

in this study, and thereafter focus on applying this

method for more skills, in neurological patients and

in natural living situations.

REFERENCES

Jain A.K., Duin R.P.W. and Mao J. (2000) Statistical

Pattern Recognition: A Review. IEEE Trans Pattern

Analysis and Machine Intelligence, 22.

Leutheuser H., Schuldhaus D. and Eskofier B.M. (2013)

Hierarchical, Multi-sensor based Classification of

Daily Life Activities: Comparison with State-of-the-

art Algorithms using a Benchmark Dataset. PlosOne, 8