THE NEED FOR IMPULSIVITY & SMOOTHNESS

Improving HCI by Qualitatively Measuring

New High-Level Human Motion Features

Barbara Mazzarino and Maurizio Mancini

Infomus Lab, DIST, Universit`a degli Studi di Genova, Italy

Keywords:

HCI.

Abstract:

The aim of this paper is to develop algorithms to measure motion features by investigating concepts which are

commonly used to describe movement characteristics in both research studies and everyday life: impulsivity

and smoothness. We also aim to implement such deﬁnitions in our developing environment VisNet and ﬁnally

test if they can effectively measure impulsivity and smoothness in the same way these characteristics are

perceived by human users.

1 INTRODUCTION

In the last few years one of the key issues of the Hu-

man Computer Interaction framework is the design

and creation of a new type of interfaces, able to adapt

HCI to human-human communication capabilities. In

this direction the ability of computers to detect the

user emotional state is becoming particularly relevant,

that is, computers must be equipped with interfaces

able to establish an Affect Sensitive interaction with

the user, in the sense deﬁned by Zeng in (Zeng et al.,

2009). Many different research activities have been

performed with this aim, see for example Affective

Computing in USA (Picard, 1997) and Kansei Infor-

mation processing in Japan (Hashimoto, 1997). Both

these areas in fact aim to design and implement ma-

chines that are able (i) to recognize user emotions, (ii)

to express emotional states, and (iii) “to have” emo-

tions. Such research objectives require skills tradi-

tionally belonging to separate disciplines, in partic-

ular computer technologies and humanistic research.

The cross-modal nature of this research area is multi-

disciplinary also from an application point of view, as

we can apply results in, e.g. rehabilitation, e-learning,

e-teaching, entertainment, performing arts and so on.

In our work we focus our attention on the ﬁrst of

the above aspects, i.e. the ability for machines to rec-

ognize the user emotional state. Psychologists, mu-

sicologists, researchers on music perception and hu-

man movement, like Wallbott & Scherer (Wallbott

and Scherer, 1986), Gallaher (Gallaher, 1992), deem

it is important in recognizing emotions the evalua-

tion of body motion qualiﬁers such as: speed, am-

plitude, energy and so on. Wallbott demonstrated in

(Wallbott, 1998) that body activity, expansiveness and

power are discriminating factors in communicating a

large number of emotional states. In a similar ap-

proach, R. Laban elaborated his Theory of Effort (La-

ban and Lawrence, 1947), in which he identify some

motion cues that are conveyor of high level informa-

tion as emotional states. Also in music perception

there are audio features responsible in communicating

emotions, such features are related to tempo, volume,

pitch, articulation, dynamics and so on.

In this paper we present a method for measur-

ing two of the features mentioned above: movement

impulsivity and smoothness. Impulsivity indicates

wether or not movement presents sudden and abrupt

changes in energy. For example, an unexpected dan-

ger like a car approaching a person crossing the street

may induce a sudden and impulsive reaction in the

person movement, due to the emotion of fear/terror.

Smoothness identiﬁes the continuity/ﬂuencyof move-

ment. Happy and relaxed persons usually communi-

cate their state by producing body movements that are

very ﬂuent and continuous. Instead, angry and tensed

persons perform quick and short body movements ex-

hibiting abrupt changes in limbs curvature/speed.

2 IMPULSIVITY DEFINITION

In this paragraph we will present the main references,

from different research areas, we used to reach impul-

Mazzarino B. and Mancini M. (2009).

THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level Human Motion Features.

In Proceedings of the International Conference on Signal Processing and Multimedia Applications, pages 62-67

DOI: 10.5220/0002232900620067

 SciTePress

sivity deﬁnition.

From physics we refer to the Impulsive Momentum

TheoremIf Force and Mass are considered as con-

stants then the following rule is respected: I = F △

t = m △ v =△ p. If the starting and the ending veloc-

ities are known then the rule above can be written as:

△ p = m(vf − vi).

The underlying concept of this theorem considers the

impulse as a variation of the momentum,i.e a pertur-

bation of the state, useful to reach a deﬁnition and to

reach a reference measure.

In psychology Impulsivity is deﬁned as “actions that

are poorly conceived, prematurely expressed, unduly

risky, or inappropriate to the situation and that often

result in undesirable outcome”.

In this context Impulsivity is an important aspect

to consider for evaluating some speciﬁc pathologies,

but, unfortunately, the evaluation is based on ques-

tionnaires. From the deﬁnition we can observe that

an impulsive behavior or gesture lacks of premedi-

tation, that is, it is performed without a signiﬁcant

preparation phase.

A good example of motion analysis for evaluating

Impulsivity, is represented by Heiser and colleagues

work (Heiser et al., 2004). Using a IR video cam-

era they recorded the motion of young subject with

Hyperkinetic Disorders before and after medicine as-

sumption, and they analysed the material with manual

annotation and single point tracking. The motion of

these subjects has been classiﬁed as “was 3.4 times as

far, covered a 3.8-fold greater area, and had a more

linear and less complex movement pattern”, that, for

our purposes, can be translate as linear, without com-

plex pattern.

Closed to our research area is the analysis of nat-

ural gestures conveying information to support ver-

bal communication. In this area there is a well de-

ﬁned taxonomy of gestures in which Beat Gestures

are similar to impulsive movements. In the Wilson et

al. (Wilson et al., 1996) work, beat gestures are de-

ﬁned as bi-phased differently from the other gestures

(deitic, metaphoric and iconic) that are tri-phased.

The identiﬁed phases are R (Rest), T (Transition) and

S (Stroke), each one characterized by different exe-

cution distance, velocity and magnitude. The phases

characterization can be used for the deﬁnition and the

evaluation of Impulsivity.

An important theory to which refer is the Effort

Theory by R. Laban, well resumed by Aliza Shapiro

(Shapiro, 1999):

“Effort is the dynamic quality or inner attitude of

movement. Laban identiﬁed four Effort Qualities in

human movement: Flow, Weight, Time, and Space.

What this means is that when a person moves, she can

be understood to move with some combination of the

above qualities. A runner might use Flow and Time. A

tap dancer might use Weight and Time. Clearly, there

is a variety of ways of tap dancing and of existing

in Weight and Time. Each Effort Quality is therefore

further reﬁned. Flow consists of a continuum from

Bound to Free. Weight Quality consists of a contin-

uum from Strong Pressure to Light. Time consists of a

continuum from Sustained to Quick. And Space con-

sists of a continuum from Direct to Multifocused. The

poles of these continua are termed elements ”.

Using the Effort Qualities we can deﬁne the impulsive

gesture as a motion characterized by a Time = sudden

and a Flow = free.

Resuming what is described above, by integrat-

ing different approaches we can deﬁne Impulsivity as

a short time perturbation of the subject motion

state. Moreover with this multidisciplinary overview

we obtain an impulsive gesture characterization that

can be resumed as gestures:

• performed without premeditation, i.e. looking to

the motion phases with a very short or absent

preparation phase.

• performed with a simple pattern, i.e. simple shape

performed.

• characterized by a T phase, i.e. short duration and

high magnitude.

• performed with Time = sudden and Flow =free in

Laban terms.

3 SMOOTHNESS DEFINITION

From English dictionary, smooth: “generally ﬂat or

unrufﬂed, as a calm sea; free from or proceeding

without abrupt curves, bends, etc.; allowing or hav-

ing an even, uninterrupted movement or ﬂow”.

In mathematics, smoothness is linked to the speed

of variation, that is, a smooth function is a function

that varies “slowly” in time; more precisely, smooth

functions are those that have derivatives of all orders.

In music smoothness corresponds to articulation in

music performance, as for example DiPaola (DiPaola

and Arya, 2004) states: “phrasing of music refers to

notes being smoothly connected (legato) or not (stac-

cato) ”.

In psychopathology, smoothness of human move-

ment could allow one to diagnose psychological dis-

orders, for example schizophrenia: patients move-

ments are described “staccato-like, jerky and angu-

lar”, while they become “smooth and rounded” after

successful therapy, as reported in (Wallbott, 1989).

As reported in the same study, smooth movements

THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level

Human Motion Features

are those:“characterized distally by large circumfer-

ence, long waylength, high mean velocity, but not

abrupt changes in velocity or acceleration (standard

deviations of velocity and acceleration); thus, smooth

movements seem to be large in terms of space and ex-

hibit a high but even velocity ”.

They contrast with “precise, angular, rigid and

hasty ”movements. Gallaher (Gallaher, 1992) refers

to smooth and ﬂuid movements: “an individual high

on this factor has a smooth voice, ﬂowing speech and

gestures, and a ﬂuid walk; such a person would ap-

pear graceful and coordinated ”.

She mainly uses the term smooth when referring

to gesture and voice characteristics, while ﬂuid is

used for the walking style. Smooth/ﬂuid movements

are often associated with slow, sluggish and lethargic

movements, in contrast with large and energetic body

movement. Slowness in movements corresponds to

the deﬁnition of smooth functions as slowly varying

functions in mathematics.

Wallbott measured displacement of hand in psy-

chiatric patients behavior and found four main move-

ment characteristics: space, which describes the ex-

tension of movement; hastiness, which is related to

speed and acceleration; intensity, which describes the

energy of a movement; ﬂuency-course, which is re-

lated to the quality between the beginning and the end

of a movement. Wallbott states that smoothness is a

possible value for the ﬂuency-course characteristics,

thus demonstrating the importance of such parameter

in describing movement quality.

The concept of movement smoothness has been

studied also in R. Laban’s Theory of Effort (Laban

and Lawrence, 1947). In Laban’s model, movement

quality is characterized by 4 components: space, rep-

resenting the way in which the movement performer

approaches space, in a direct, single-focused way or

in a ﬂexible, multi-focused way; weight, describing

movement impact, that is, whether it expresses less or

more energy; time, modeling how movement appears,

for example suddenly or in a prepared way, lasting

a long time; ﬂow, expressing the quantity of control

the performer has over its movements, e.g., one can

fully control its movements or let movement and en-

ergy ﬂow through its body freely.

Different movement qualities correspond to different

values combinations for the Laban’s parameters: for

example punching is usually direct, strong and sud-

den; ﬂoating is indirect, light and sustained. Smooth

movements, as reported in (Newlove, 2007), are usu-

ally direct, light, sustained and bound.

4 ALGORITHMS

We now aim to formally deﬁne and implement algo-

rithms for extracting impulsivity and smoothness of a

human performer in realtime and from a video source.

4.1 Impulsivity

Our aim, after reached the deﬁnition, was to develop

an algorithm for the automatic evaluation of impul-

sivity. In this paper we present preliminary results

of the algorithm which works in semi-realtime (since

this measure can be performed at the end of the ges-

ture and not during the motion). For the gesture iden-

tiﬁcation gesture execution in time we use a motion

segmentation based on the Quantity of Motion (Ca-

murri et al., 2004).

The duration of the gesture has to respect the limits

highlighted in the deﬁnition, i.e. to be “fast”.

In our context the most important factor is the fast

attack of the gesture and not only its short duration.

In order to quantify the attack we start considering

the premeditation and the reaction time. For exam-

ple in athletics the rules of the International Associa-

tion of Athletics Federations ﬁxed the minimal reac-

tion time to 0.1sec (less is considered a false start),

because it considers that the time interval between a

sound signal and the voluntary motor activation in a

normal subject is around 140-160 milliseconds. In-

cluding this consideration in our case, we set the start-

ing phase of a gesture to be faster of a voluntary re-

action, i.e. ≤ 0.15sec. The empirical value we found,

during our tests, for the impulsive gesture time dura-

tion is dt = 0.45sec.

Since we are interested in gestures with “high magni-

tude ”we considered only gestures with high energy,

so the threshold used for the segmentation has been

ﬁxed to assume an empirically high value with respect

to the standard one.

The impulsive gesture is deﬁned with respect to the

current activity, to do this we considered a perturba-

tion as a fast (as above described) modiﬁcation of the

current motion, and we did it by evaluating the usage

of the space occupation. With empirically considera-

tions, in order to modify rapidly the actual motion, it

is necessary to modify rapidly the posture and in par-

ticular to perform a modiﬁcation of space occupation.

For this evaluation we use the variation of the space

(Camurri et al., 2004) in the time window of the ges-

ture duration DCI.

The global applied algorithm can then be written as:

let

△ t = 0.45sec

;

let gesture

threshold = 0.02

;

(energy ≥ threshold)

SIGMAP 2009 - International Conference on Signal Processing and Multimedia Applications

evaluate the GestureTimeDuration

;

dt ≥ 0

dt ≤△ t

then

ImpulsivityIndex =△ CI/dt

;

4.2 Smoothness

Research in (Todorov and Jordan, 1998) demonstrates

a correspondence between (i) smooth trajectories per-

formed by human arms, (ii) minimization of the third-

order derivative of the hand position (called jerk in

physics) and (iii) correlation between hand trajectory

curvature and velocity. In our work we use an ap-

proach similar to (iii) to determine if a trajectory is

smooth or not starting from the trajectory curvature

and velocity.

Let us ﬁrst explain how the input data is pre-

processed: the input to our system consists of video

frames frames at 60 Hz showing a moving person.

During the preprocessing phase, for each video frame

the system extracts the 2D position (x,y) of the

barycenter of a green marker placed on the person

right or left hand and stores it in a buffer consisting of

60 samples, while the oldest element of the buffer is

discarded. The hand position buffer is then provided

as inputto the smoothnesscomputation algorithm: for

every sample (x,y) in the buffer we compute curva-

ture k and velocity v as:

k(x,y) =



′

′′

− y

′

′′

′2

+ y

′2

)



v(x,y) =

′2

+ y

′2

(1)

where x

′

, y

′

, x

′′

and y

′′

are the ﬁrst and second or-

der derivatives of x and y. To compute them from

the buffer of samples (x,y) we apply the Savitzky-

Golay ﬁlter (Savitzky and Golay, 1964) that provides

as output both the ﬁltered signal and an approxima-

tion of the n−th order smoothed derivatives. As men-

tioned above, we deﬁne our algorithm for comput-

ing smoothness by taking inspiration from (Todorov

and Jordan, 1998), that is, we compute correlation be-

tween trajectory curvature and velocity. We consider

the Pearson correlation coefﬁcient for two variables,

that is, in our algorithm, log(k) and log(v):

ρ(k,v) =

log(k),log(v)

log(k)

log(v)

(2)

However, k and v are computed over a “short” time

window, so we could approximate the covariance

log(k),log(v)

with 1, as the k and v variate (or not) ap-

proximately at the same time:

′

(k,v) =

log(k)

log(v)

(3)

If we now apply the above steps on the buffer of sam-

ples representing the user hand trajectory we could

obtain a vector s of real numbers corresponding to the

trajectory Smoothness Index:

let

SmoothnessIndex =

empty vector;

for every

(x,y) ∈

input buffer of samples

compute

k(x,y)

v(x,y)

and

′

(k,v)

;

insert

′

(k,v)

SmoothnessIndex

;

endfor

5 PILOT EXPERIMENTS &

RESULTS

We have conducted preliminary studies inside the

EyesWeb developing environment, a system we cre-

ated to allow researchers and normal users to visually

build applications involving multimodal input, com-

putation and output (Camurri et al., 2004). The ﬁnal

aim of these studies will be to determine if our deﬁ-

nition of impulsivity and smoothness matches or not

the human user perception of these cues. We present

preliminary studies in which we mainly test if move-

ments performed intentionally with different impul-

sivity and smoothness by a human user are classiﬁed

by our algorithms in the intended way.

5.1 Measuring Impulsivity

Setup and Procedure. The analysis has been ap-

plied on bi-dimensional motion performed in front of

the video Camera. Subjects are required to perform

impulsive gestures after a period of motion or non-

motion for cognitive saturation purposes.

The non compressed video signal (60p, 1280x720,

BGR) has been processed with EyesWeb software

platform to extract the motion features described in

Section 4.1. Brieﬂy an algorithm for the background

subtraction has been applyed to the video input in or-

der to extract the Silhouette of the subject. From the

Silhouette the space occupation and the energy of the

motion has been evaluated to identify gestures (see in

Figure 1 an example) and to calculate the Impulsivity.

Results and Discussion. The EyesWeb software

platform to support the development of real-time mul-

timodal distributed interactive applications. Is a vi-

sual environment with predeﬁned sw modules, e.g.

for the real-time evaluation of low level motion fea-

ture. Using this platform, we implemented the pro-

posed formula and perfomed some tests. In Figure 2

is represented a snapshot of the real-time extraction of

the Impulsivity Index. At this stage of our experiment

THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level

Human Motion Features

Figure 1: This is an off line representation of the

energy motion feature with respect to the threshold

value. In the green circle there is the motion bell re-

lated to the impulsive gesture. It is important to notice

that the motion bell of this gesture, is isolated with

respect to the other bells, i.e. is a perturbation of the

current state.

the index gives an indication of the performed gesture

at the end of the execution because it needs to know

the time duration of the gesture.

Figure 2: This is a snapshot of the software platform

during a real-time evaluation. The Impulsivity Index,

in the bottom left part of the image, reach its maxi-

mum at the end of the gesture. On the bottom right

part of the image there is the related Contraction In-

dex graph, that measure the posture modiﬁcations. In

the centre there is the current view of the camera.

Results show that our algorithm is able to iden-

tify an impulsive gesture, following the deﬁnition of

Section 2, given a high value of the Impulsivity In-

dex. The quantiﬁcation of this high value has to been

reﬁne, at this moment the impulsive gestures are iden-

tify by the Impulsivity Index maximum.

5.2 Measuring Smoothness

Setup and Procedure. With this experiment we

aim to verify whether the algorithm presented in Sec-

tion 4.2 is able to recognize movements performed in

a smooth way. As a preliminary test we instructed

the performer to produce movements with the follow-

ing intentions: (A) a circular movement, trying to be

as smooth as possible, that is, by maintaining a con-

stant speed and curvature; (B) a squared movement,

produced by performing sharp direction variations in

the square vertices; (C) a linear horizontal movement,

performed by maintaining a constant speed; (D) a lin-

ear horizontal movement presenting interruptions.

That is, movements A and C presented a high level

of smoothness while movements B and D presented

sharp variations and segmentation, thus they are not

smooth. This is the way we expected our algorithm

classifying these four movements.

Results and Discussion. Results of this preliminary

study are reported in Figures 3 and 4, corresponding

respectively to movements A and B and movements

C and D described in the previous Section. The up-

per part of each Figure shows the trajectories of the

performer hand: continuous line represents smooth

movements (constant speed) while segments and dots

represent movements with sharp direction variations

or segmentation. The bottom part of Figures reports

the information provided as output by our system

EyesWeb in realtime: the trajectory as it was detected

by the program and the trajectory Smoothness Index

computed between log(k) and log(v), as described in

Section 4.2.

Figure 3: Circular and square trajectories: Smoothness

Index is high when computed on the circular smooth

trajectory (left) and is approximately zero when com-

puted on the square non-continuous one (right).

Results show that our algorithm is able to distin-

guish between movements performed smoothly and

movements performed with sharp direction varia-

tions. As we expected the ﬁrst class of movements

(A and C) present a high Pearson coefﬁcient while for

the second class of movements (B and D) the coef-

ﬁcient drops to approximately zero. Of course, these

preliminary results do not demonstrate the correctness

of our algorithm and further more sophisticated tests

should be performed in future.

SIGMAP 2009 - International Conference on Signal Processing and Multimedia Applications

Figure 4: Linear trajectories: Smoothness Index is

high when movement is smooth and continuous (left)

and is low when computed on the interrupted move-

ment (right).

6 CONCLUSIONS

Improving human computer interaction by detecting

the user emotional state is a relevant research topic.

Humanistic research shows the possibility to identify

emotional states by analyzing body motion feature.

The work presented in this paper aims to develop al-

gorithms to measure two motion features: impulsivity

and smoothness. In the paper we described the deﬁni-

tions of these two features, the algorithms to evaluate

them in real time and the methods we developed to

test them.

Future improvements will be designed after new

experimental sessions, using both recorded and live

performances. Initially we plan to test our algo-

rithms on a video corpus consisting of several ges-

tures recorded in our lab. The gestures are performed

by student and professional dancers, and martial art

experts. All the videos included in this corpus have

been annotated and rated by experts. These ratings

will guide us in tuning and reﬁning our methodology.

The reﬁned algorithmswill then be applied in live per-

formances combined with the extraction of other mo-

tion features we already implemented and validated

(for example Quantity of Motion and Contraction In-

dex) in order to detect the performer emotional inten-

tion. At the same time, we plan to continue validating

our algorithms by asking subjects to rate the video

corpus described above. Performed and future work

are addressed in the framework of the EUICT Project

SAME (www.sameproject.eu) and the EU Culture

2007 project CoMeDiA (www.comedia.eu.org).

REFERENCES

Camurri, A., Mazzarino, B., and Volpe, G. (2004). Anal-

ysis of expressive gesture: The eyes web expressive

gesture processing library. Lecture notes in computer

science.

DiPaola, S. and Arya, A. (2004). Affective communication

remapping in musicface system. In Electronic Imag-

ing & Visual Arts.

Gallaher, P. E. (1992). Individual differences in nonverbal

behavior: Dimensions of style. Journal of Personality

and Social Psychology, 63(1):133–145.

Hashimoto, S. (1997). Kansei as the third target of infor-

mation processing and related topics in japan. In Pro-

ceedings of the International Workshop on KANSEI:

The technology of emotion, pages 101–104.

Heiser, P., Frey, J., Smidt, J., Sommerlad, C., M.Wehmeier,

P., J.Hebebrand, and Remschmidt, H. (2004). Ob-

jective measurement of hyperactivity, impulsivity, and

inattention in children with hyperkinetic disorders be-

fore and after treatment with methylphenidate. Euro-

pean Child & Adolescent Psychiatry, 13(2):100–104.

Laban, R. and Lawrence, F. C. (1947). Effort. Macdonald

& Evans, USA.

Newlove, J. (2007). Laban for Actors and Dancers: Putting

Laban’s Movement Theory Into Practice : a Step-by-

step Guide. Nick Hern Books, UK.

Picard, R. (1997). Affective Computing. MIT Press.

Savitzky, A. and Golay, M. J. E. (1964). Smoothing and

differentiation of data by simpliﬁed least squares pro-

cedures. Analytical chemistry, 36(8):1627–1639.

Shapiro, A. I. (1999). The Movement Phrase and its clin-

ical value in Dance/Movement Therapy. PhD thesis,

Master of Arts in Dance/Movement Therapy, MCP-

Hahnemann University.

Todorov, E. and Jordan, M. I. (1998). Smoothness maxi-

mization along a predeﬁned path accurately predicts

the speed proﬁles of complex arm movements. Jour-

nal of Neurophysiology, 80(2):696–714.

Wallbott, H. G. (1989). Movement quality changes in psy-

chopathological disorders. Normalities and Abnor-

malities in Human Movement. Medicine and Sport

Science, 29:128–146.

Wallbott, H. G. (1998). Bodily expression of emotion. Eu-

ropean Journal of Social Psychology, 28:879–896.

Wallbott, H. G. and Scherer, K. R. (1986). Cues and chan-

nels in emotion recognition. Journal of Personality

and Social Psychology, 51(4):690–699.

Wilson, A., Bobick, A., and Cassell, J. (1996). Recovering

the temporal structure of natural gesture. In Proc. of

the Second Intern. Conf. on Automatic Face and Ges-

ture Recognition.

Zeng, Z., Pantic, M., Roisman, G., and Huang, T. (2009).

A survey of affect recognition methods: audio, visual

and spontaneous expressions. IEEE Transactions on

Pattern Analysis and Machine Intelligence, 31(1).

THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level

Human Motion Features