THE NEED FOR IMPULSIVITY & SMOOTHNESS
Improving HCI by Qualitatively Measuring
New High-Level Human Motion Features
Barbara Mazzarino and Maurizio Mancini
Infomus Lab, DIST, Universit`a degli Studi di Genova, Italy
Keywords:
HCI.
Abstract:
The aim of this paper is to develop algorithms to measure motion features by investigating concepts which are
commonly used to describe movement characteristics in both research studies and everyday life: impulsivity
and smoothness. We also aim to implement such definitions in our developing environment VisNet and finally
test if they can effectively measure impulsivity and smoothness in the same way these characteristics are
perceived by human users.
1 INTRODUCTION
In the last few years one of the key issues of the Hu-
man Computer Interaction framework is the design
and creation of a new type of interfaces, able to adapt
HCI to human-human communication capabilities. In
this direction the ability of computers to detect the
user emotional state is becoming particularly relevant,
that is, computers must be equipped with interfaces
able to establish an Affect Sensitive interaction with
the user, in the sense defined by Zeng in (Zeng et al.,
2009). Many different research activities have been
performed with this aim, see for example Affective
Computing in USA (Picard, 1997) and Kansei Infor-
mation processing in Japan (Hashimoto, 1997). Both
these areas in fact aim to design and implement ma-
chines that are able (i) to recognize user emotions, (ii)
to express emotional states, and (iii) “to have” emo-
tions. Such research objectives require skills tradi-
tionally belonging to separate disciplines, in partic-
ular computer technologies and humanistic research.
The cross-modal nature of this research area is multi-
disciplinary also from an application point of view, as
we can apply results in, e.g. rehabilitation, e-learning,
e-teaching, entertainment, performing arts and so on.
In our work we focus our attention on the first of
the above aspects, i.e. the ability for machines to rec-
ognize the user emotional state. Psychologists, mu-
sicologists, researchers on music perception and hu-
man movement, like Wallbott & Scherer (Wallbott
and Scherer, 1986), Gallaher (Gallaher, 1992), deem
it is important in recognizing emotions the evalua-
tion of body motion qualifiers such as: speed, am-
plitude, energy and so on. Wallbott demonstrated in
(Wallbott, 1998) that body activity, expansiveness and
power are discriminating factors in communicating a
large number of emotional states. In a similar ap-
proach, R. Laban elaborated his Theory of Effort (La-
ban and Lawrence, 1947), in which he identify some
motion cues that are conveyor of high level informa-
tion as emotional states. Also in music perception
there are audio features responsible in communicating
emotions, such features are related to tempo, volume,
pitch, articulation, dynamics and so on.
In this paper we present a method for measur-
ing two of the features mentioned above: movement
impulsivity and smoothness. Impulsivity indicates
wether or not movement presents sudden and abrupt
changes in energy. For example, an unexpected dan-
ger like a car approaching a person crossing the street
may induce a sudden and impulsive reaction in the
person movement, due to the emotion of fear/terror.
Smoothness identifies the continuity/fluencyof move-
ment. Happy and relaxed persons usually communi-
cate their state by producing body movements that are
very fluent and continuous. Instead, angry and tensed
persons perform quick and short body movements ex-
hibiting abrupt changes in limbs curvature/speed.
2 IMPULSIVITY DEFINITION
In this paragraph we will present the main references,
from different research areas, we used to reach impul-
62
Mazzarino B. and Mancini M. (2009).
THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level Human Motion Features.
In Proceedings of the International Conference on Signal Processing and Multimedia Applications, pages 62-67
DOI: 10.5220/0002232900620067
Copyright
c
SciTePress
sivity definition.
From physics we refer to the Impulsive Momentum
TheoremIf Force and Mass are considered as con-
stants then the following rule is respected: I = F
t = m v = p. If the starting and the ending veloc-
ities are known then the rule above can be written as:
p = m(vf vi).
The underlying concept of this theorem considers the
impulse as a variation of the momentum,i.e a pertur-
bation of the state, useful to reach a definition and to
reach a reference measure.
In psychology Impulsivity is defined as “actions that
are poorly conceived, prematurely expressed, unduly
risky, or inappropriate to the situation and that often
result in undesirable outcome”.
In this context Impulsivity is an important aspect
to consider for evaluating some specific pathologies,
but, unfortunately, the evaluation is based on ques-
tionnaires. From the definition we can observe that
an impulsive behavior or gesture lacks of premedi-
tation, that is, it is performed without a significant
preparation phase.
A good example of motion analysis for evaluating
Impulsivity, is represented by Heiser and colleagues
work (Heiser et al., 2004). Using a IR video cam-
era they recorded the motion of young subject with
Hyperkinetic Disorders before and after medicine as-
sumption, and they analysed the material with manual
annotation and single point tracking. The motion of
these subjects has been classified as “was 3.4 times as
far, covered a 3.8-fold greater area, and had a more
linear and less complex movement pattern”, that, for
our purposes, can be translate as linear, without com-
plex pattern.
Closed to our research area is the analysis of nat-
ural gestures conveying information to support ver-
bal communication. In this area there is a well de-
fined taxonomy of gestures in which Beat Gestures
are similar to impulsive movements. In the Wilson et
al. (Wilson et al., 1996) work, beat gestures are de-
fined as bi-phased differently from the other gestures
(deitic, metaphoric and iconic) that are tri-phased.
The identified phases are R (Rest), T (Transition) and
S (Stroke), each one characterized by different exe-
cution distance, velocity and magnitude. The phases
characterization can be used for the definition and the
evaluation of Impulsivity.
An important theory to which refer is the Effort
Theory by R. Laban, well resumed by Aliza Shapiro
(Shapiro, 1999):
“Effort is the dynamic quality or inner attitude of
movement. Laban identified four Effort Qualities in
human movement: Flow, Weight, Time, and Space.
What this means is that when a person moves, she can
be understood to move with some combination of the
above qualities. A runner might use Flow and Time. A
tap dancer might use Weight and Time. Clearly, there
is a variety of ways of tap dancing and of existing
in Weight and Time. Each Effort Quality is therefore
further refined. Flow consists of a continuum from
Bound to Free. Weight Quality consists of a contin-
uum from Strong Pressure to Light. Time consists of a
continuum from Sustained to Quick. And Space con-
sists of a continuum from Direct to Multifocused. The
poles of these continua are termed elements ”.
Using the Effort Qualities we can define the impulsive
gesture as a motion characterized by a Time = sudden
and a Flow = free.
Resuming what is described above, by integrat-
ing different approaches we can define Impulsivity as
a short time perturbation of the subject motion
state. Moreover with this multidisciplinary overview
we obtain an impulsive gesture characterization that
can be resumed as gestures:
performed without premeditation, i.e. looking to
the motion phases with a very short or absent
preparation phase.
performed with a simple pattern, i.e. simple shape
performed.
characterized by a T phase, i.e. short duration and
high magnitude.
performed with Time = sudden and Flow =free in
Laban terms.
3 SMOOTHNESS DEFINITION
From English dictionary, smooth: “generally flat or
unruffled, as a calm sea; free from or proceeding
without abrupt curves, bends, etc.; allowing or hav-
ing an even, uninterrupted movement or flow”.
In mathematics, smoothness is linked to the speed
of variation, that is, a smooth function is a function
that varies “slowly” in time; more precisely, smooth
functions are those that have derivatives of all orders.
In music smoothness corresponds to articulation in
music performance, as for example DiPaola (DiPaola
and Arya, 2004) states: “phrasing of music refers to
notes being smoothly connected (legato) or not (stac-
cato) ”.
In psychopathology, smoothness of human move-
ment could allow one to diagnose psychological dis-
orders, for example schizophrenia: patients move-
ments are described “staccato-like, jerky and angu-
lar”, while they become “smooth and rounded” after
successful therapy, as reported in (Wallbott, 1989).
As reported in the same study, smooth movements
THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level
Human Motion Features
63
are those:“characterized distally by large circumfer-
ence, long waylength, high mean velocity, but not
abrupt changes in velocity or acceleration (standard
deviations of velocity and acceleration); thus, smooth
movements seem to be large in terms of space and ex-
hibit a high but even velocity ”.
They contrast with “precise, angular, rigid and
hasty ”movements. Gallaher (Gallaher, 1992) refers
to smooth and fluid movements: “an individual high
on this factor has a smooth voice, flowing speech and
gestures, and a fluid walk; such a person would ap-
pear graceful and coordinated .
She mainly uses the term smooth when referring
to gesture and voice characteristics, while fluid is
used for the walking style. Smooth/fluid movements
are often associated with slow, sluggish and lethargic
movements, in contrast with large and energetic body
movement. Slowness in movements corresponds to
the definition of smooth functions as slowly varying
functions in mathematics.
Wallbott measured displacement of hand in psy-
chiatric patients behavior and found four main move-
ment characteristics: space, which describes the ex-
tension of movement; hastiness, which is related to
speed and acceleration; intensity, which describes the
energy of a movement; fluency-course, which is re-
lated to the quality between the beginning and the end
of a movement. Wallbott states that smoothness is a
possible value for the fluency-course characteristics,
thus demonstrating the importance of such parameter
in describing movement quality.
The concept of movement smoothness has been
studied also in R. Laban’s Theory of Effort (Laban
and Lawrence, 1947). In Laban’s model, movement
quality is characterized by 4 components: space, rep-
resenting the way in which the movement performer
approaches space, in a direct, single-focused way or
in a flexible, multi-focused way; weight, describing
movement impact, that is, whether it expresses less or
more energy; time, modeling how movement appears,
for example suddenly or in a prepared way, lasting
a long time; flow, expressing the quantity of control
the performer has over its movements, e.g., one can
fully control its movements or let movement and en-
ergy flow through its body freely.
Different movement qualities correspond to different
values combinations for the Laban’s parameters: for
example punching is usually direct, strong and sud-
den; floating is indirect, light and sustained. Smooth
movements, as reported in (Newlove, 2007), are usu-
ally direct, light, sustained and bound.
4 ALGORITHMS
We now aim to formally define and implement algo-
rithms for extracting impulsivity and smoothness of a
human performer in realtime and from a video source.
4.1 Impulsivity
Our aim, after reached the definition, was to develop
an algorithm for the automatic evaluation of impul-
sivity. In this paper we present preliminary results
of the algorithm which works in semi-realtime (since
this measure can be performed at the end of the ges-
ture and not during the motion). For the gesture iden-
tification gesture execution in time we use a motion
segmentation based on the Quantity of Motion (Ca-
murri et al., 2004).
The duration of the gesture has to respect the limits
highlighted in the definition, i.e. to be “fast”.
In our context the most important factor is the fast
attack of the gesture and not only its short duration.
In order to quantify the attack we start considering
the premeditation and the reaction time. For exam-
ple in athletics the rules of the International Associa-
tion of Athletics Federations fixed the minimal reac-
tion time to 0.1sec (less is considered a false start),
because it considers that the time interval between a
sound signal and the voluntary motor activation in a
normal subject is around 140-160 milliseconds. In-
cluding this consideration in our case, we set the start-
ing phase of a gesture to be faster of a voluntary re-
action, i.e. 0.15sec. The empirical value we found,
during our tests, for the impulsive gesture time dura-
tion is dt = 0.45sec.
Since we are interested in gestures with “high magni-
tude ”we considered only gestures with high energy,
so the threshold used for the segmentation has been
fixed to assume an empirically high value with respect
to the standard one.
The impulsive gesture is defined with respect to the
current activity, to do this we considered a perturba-
tion as a fast (as above described) modification of the
current motion, and we did it by evaluating the usage
of the space occupation. With empirically considera-
tions, in order to modify rapidly the actual motion, it
is necessary to modify rapidly the posture and in par-
ticular to perform a modification of space occupation.
For this evaluation we use the variation of the space
(Camurri et al., 2004) in the time window of the ges-
ture duration DCI.
The global applied algorithm can then be written as:
let
t = 0.45sec
;
let gesture
threshold = 0.02
;
if
(energy threshold)
SIGMAP 2009 - International Conference on Signal Processing and Multimedia Applications
64
evaluate the GestureTimeDuration
dt
;
If
dt 0
if
dt t
then
ImpulsivityIndex = CI/dt
;
4.2 Smoothness
Research in (Todorov and Jordan, 1998) demonstrates
a correspondence between (i) smooth trajectories per-
formed by human arms, (ii) minimization of the third-
order derivative of the hand position (called jerk in
physics) and (iii) correlation between hand trajectory
curvature and velocity. In our work we use an ap-
proach similar to (iii) to determine if a trajectory is
smooth or not starting from the trajectory curvature
and velocity.
Let us first explain how the input data is pre-
processed: the input to our system consists of video
frames frames at 60 Hz showing a moving person.
During the preprocessing phase, for each video frame
the system extracts the 2D position (x,y) of the
barycenter of a green marker placed on the person
right or left hand and stores it in a buffer consisting of
60 samples, while the oldest element of the buffer is
discarded. The hand position buffer is then provided
as inputto the smoothnesscomputation algorithm: for
every sample (x,y) in the buffer we compute curva-
ture k and velocity v as:
k(x,y) =
x
y
′′
y
x
′′
(x
2
+ y
2
)
3
2
v(x,y) =
p
x
2
+ y
2
(1)
where x
, y
, x
′′
and y
′′
are the first and second or-
der derivatives of x and y. To compute them from
the buffer of samples (x,y) we apply the Savitzky-
Golay filter (Savitzky and Golay, 1964) that provides
as output both the filtered signal and an approxima-
tion of the nth order smoothed derivatives. As men-
tioned above, we define our algorithm for comput-
ing smoothness by taking inspiration from (Todorov
and Jordan, 1998), that is, we compute correlation be-
tween trajectory curvature and velocity. We consider
the Pearson correlation coefficient for two variables,
that is, in our algorithm, log(k) and log(v):
ρ(k,v) =
σ
log(k),log(v)
σ
log(k)
σ
log(v)
(2)
However, k and v are computed over a “short” time
window, so we could approximate the covariance
σ
log(k),log(v)
with 1, as the k and v variate (or not) ap-
proximately at the same time:
ρ
(k,v) =
1
σ
log(k)
σ
log(v)
(3)
If we now apply the above steps on the buffer of sam-
ples representing the user hand trajectory we could
obtain a vector s of real numbers corresponding to the
trajectory Smoothness Index:
let
SmoothnessIndex =
empty vector;
for every
(x,y)
input buffer of samples
compute
k(x,y)
,
v(x,y)
and
ρ
(k,v)
;
insert
ρ
(k,v)
in
SmoothnessIndex
;
endfor
5 PILOT EXPERIMENTS &
RESULTS
We have conducted preliminary studies inside the
EyesWeb developing environment, a system we cre-
ated to allow researchers and normal users to visually
build applications involving multimodal input, com-
putation and output (Camurri et al., 2004). The final
aim of these studies will be to determine if our defi-
nition of impulsivity and smoothness matches or not
the human user perception of these cues. We present
preliminary studies in which we mainly test if move-
ments performed intentionally with different impul-
sivity and smoothness by a human user are classified
by our algorithms in the intended way.
5.1 Measuring Impulsivity
Setup and Procedure. The analysis has been ap-
plied on bi-dimensional motion performed in front of
the video Camera. Subjects are required to perform
impulsive gestures after a period of motion or non-
motion for cognitive saturation purposes.
The non compressed video signal (60p, 1280x720,
BGR) has been processed with EyesWeb software
platform to extract the motion features described in
Section 4.1. Briefly an algorithm for the background
subtraction has been applyed to the video input in or-
der to extract the Silhouette of the subject. From the
Silhouette the space occupation and the energy of the
motion has been evaluated to identify gestures (see in
Figure 1 an example) and to calculate the Impulsivity.
Results and Discussion. The EyesWeb software
platform to support the development of real-time mul-
timodal distributed interactive applications. Is a vi-
sual environment with predefined sw modules, e.g.
for the real-time evaluation of low level motion fea-
ture. Using this platform, we implemented the pro-
posed formula and perfomed some tests. In Figure 2
is represented a snapshot of the real-time extraction of
the Impulsivity Index. At this stage of our experiment
THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level
Human Motion Features
65
Figure 1: This is an off line representation of the
energy motion feature with respect to the threshold
value. In the green circle there is the motion bell re-
lated to the impulsive gesture. It is important to notice
that the motion bell of this gesture, is isolated with
respect to the other bells, i.e. is a perturbation of the
current state.
the index gives an indication of the performed gesture
at the end of the execution because it needs to know
the time duration of the gesture.
Figure 2: This is a snapshot of the software platform
during a real-time evaluation. The Impulsivity Index,
in the bottom left part of the image, reach its maxi-
mum at the end of the gesture. On the bottom right
part of the image there is the related Contraction In-
dex graph, that measure the posture modifications. In
the centre there is the current view of the camera.
Results show that our algorithm is able to iden-
tify an impulsive gesture, following the definition of
Section 2, given a high value of the Impulsivity In-
dex. The quantification of this high value has to been
refine, at this moment the impulsive gestures are iden-
tify by the Impulsivity Index maximum.
5.2 Measuring Smoothness
Setup and Procedure. With this experiment we
aim to verify whether the algorithm presented in Sec-
tion 4.2 is able to recognize movements performed in
a smooth way. As a preliminary test we instructed
the performer to produce movements with the follow-
ing intentions: (A) a circular movement, trying to be
as smooth as possible, that is, by maintaining a con-
stant speed and curvature; (B) a squared movement,
produced by performing sharp direction variations in
the square vertices; (C) a linear horizontal movement,
performed by maintaining a constant speed; (D) a lin-
ear horizontal movement presenting interruptions.
That is, movements A and C presented a high level
of smoothness while movements B and D presented
sharp variations and segmentation, thus they are not
smooth. This is the way we expected our algorithm
classifying these four movements.
Results and Discussion. Results of this preliminary
study are reported in Figures 3 and 4, corresponding
respectively to movements A and B and movements
C and D described in the previous Section. The up-
per part of each Figure shows the trajectories of the
performer hand: continuous line represents smooth
movements (constant speed) while segments and dots
represent movements with sharp direction variations
or segmentation. The bottom part of Figures reports
the information provided as output by our system
EyesWeb in realtime: the trajectory as it was detected
by the program and the trajectory Smoothness Index
computed between log(k) and log(v), as described in
Section 4.2.
Figure 3: Circular and square trajectories: Smoothness
Index is high when computed on the circular smooth
trajectory (left) and is approximately zero when com-
puted on the square non-continuous one (right).
Results show that our algorithm is able to distin-
guish between movements performed smoothly and
movements performed with sharp direction varia-
tions. As we expected the first class of movements
(A and C) present a high Pearson coefficient while for
the second class of movements (B and D) the coef-
ficient drops to approximately zero. Of course, these
preliminary results do not demonstrate the correctness
of our algorithm and further more sophisticated tests
should be performed in future.
SIGMAP 2009 - International Conference on Signal Processing and Multimedia Applications
66
Figure 4: Linear trajectories: Smoothness Index is
high when movement is smooth and continuous (left)
and is low when computed on the interrupted move-
ment (right).
6 CONCLUSIONS
Improving human computer interaction by detecting
the user emotional state is a relevant research topic.
Humanistic research shows the possibility to identify
emotional states by analyzing body motion feature.
The work presented in this paper aims to develop al-
gorithms to measure two motion features: impulsivity
and smoothness. In the paper we described the defini-
tions of these two features, the algorithms to evaluate
them in real time and the methods we developed to
test them.
Future improvements will be designed after new
experimental sessions, using both recorded and live
performances. Initially we plan to test our algo-
rithms on a video corpus consisting of several ges-
tures recorded in our lab. The gestures are performed
by student and professional dancers, and martial art
experts. All the videos included in this corpus have
been annotated and rated by experts. These ratings
will guide us in tuning and refining our methodology.
The refined algorithmswill then be applied in live per-
formances combined with the extraction of other mo-
tion features we already implemented and validated
(for example Quantity of Motion and Contraction In-
dex) in order to detect the performer emotional inten-
tion. At the same time, we plan to continue validating
our algorithms by asking subjects to rate the video
corpus described above. Performed and future work
are addressed in the framework of the EUICT Project
SAME (www.sameproject.eu) and the EU Culture
2007 project CoMeDiA (www.comedia.eu.org).
REFERENCES
Camurri, A., Mazzarino, B., and Volpe, G. (2004). Anal-
ysis of expressive gesture: The eyes web expressive
gesture processing library. Lecture notes in computer
science.
DiPaola, S. and Arya, A. (2004). Affective communication
remapping in musicface system. In Electronic Imag-
ing & Visual Arts.
Gallaher, P. E. (1992). Individual differences in nonverbal
behavior: Dimensions of style. Journal of Personality
and Social Psychology, 63(1):133–145.
Hashimoto, S. (1997). Kansei as the third target of infor-
mation processing and related topics in japan. In Pro-
ceedings of the International Workshop on KANSEI:
The technology of emotion, pages 101–104.
Heiser, P., Frey, J., Smidt, J., Sommerlad, C., M.Wehmeier,
P., J.Hebebrand, and Remschmidt, H. (2004). Ob-
jective measurement of hyperactivity, impulsivity, and
inattention in children with hyperkinetic disorders be-
fore and after treatment with methylphenidate. Euro-
pean Child & Adolescent Psychiatry, 13(2):100–104.
Laban, R. and Lawrence, F. C. (1947). Effort. Macdonald
& Evans, USA.
Newlove, J. (2007). Laban for Actors and Dancers: Putting
Laban’s Movement Theory Into Practice : a Step-by-
step Guide. Nick Hern Books, UK.
Picard, R. (1997). Affective Computing. MIT Press.
Savitzky, A. and Golay, M. J. E. (1964). Smoothing and
differentiation of data by simplified least squares pro-
cedures. Analytical chemistry, 36(8):1627–1639.
Shapiro, A. I. (1999). The Movement Phrase and its clin-
ical value in Dance/Movement Therapy. PhD thesis,
Master of Arts in Dance/Movement Therapy, MCP-
Hahnemann University.
Todorov, E. and Jordan, M. I. (1998). Smoothness maxi-
mization along a predefined path accurately predicts
the speed profiles of complex arm movements. Jour-
nal of Neurophysiology, 80(2):696–714.
Wallbott, H. G. (1989). Movement quality changes in psy-
chopathological disorders. Normalities and Abnor-
malities in Human Movement. Medicine and Sport
Science, 29:128–146.
Wallbott, H. G. (1998). Bodily expression of emotion. Eu-
ropean Journal of Social Psychology, 28:879–896.
Wallbott, H. G. and Scherer, K. R. (1986). Cues and chan-
nels in emotion recognition. Journal of Personality
and Social Psychology, 51(4):690–699.
Wilson, A., Bobick, A., and Cassell, J. (1996). Recovering
the temporal structure of natural gesture. In Proc. of
the Second Intern. Conf. on Automatic Face and Ges-
ture Recognition.
Zeng, Z., Pantic, M., Roisman, G., and Huang, T. (2009).
A survey of affect recognition methods: audio, visual
and spontaneous expressions. IEEE Transactions on
Pattern Analysis and Machine Intelligence, 31(1).
THE NEED FOR IMPULSIVITY & SMOOTHNESS - Improving HCI by Qualitatively Measuring New High-Level
Human Motion Features
67