Multimodal Analysis for Behavioural Recognition in Tele-assistance

Applications

Sorin Soviany and Sorin Puscoci

Communication Terminals ad Telematics, National Communications Studies and Research Institute,

Bucharest, Romania

Keywords: Tele-assistance, Behavioural Pattern, Hierarchical Classifier, Multimodal Analysis.

Abstract: The paper proposes an approach for behavioural recognition in which the individual conditions are

recognized using a multimodal analysis method. This approach is an extension of our previously defined

multimodal analysis method for biometrics; in this case the target application is the accurate recognition of

human behaviour in smart home environments, with main focus in the home tele-assistance integrated

services for elderly people. The proposed multimodal analysis method uses a hierarchical approach for data

classification together with a fusion rule to combine the matching scores for several behavioural patterns.

The approach novelty is given by the hierarchical classification design which provides an optimal

performance-cost trade-off for the behavioural recognition system. This optimization could be done at run-

time in practical applications.

1 INTRODUCTION

At the European level there are serious concerns

about the population ageing. According to Ageing

Report (EC, 2012) the every 3

European citizen

will be over 65 years old till 2060. The working

people vs. the 'inactive' others ratio is expected to

increase from 4:1 currently to 2:1 until 2060.

The population ageing deals with the following

issues:

 The elderly people are more likely prone

to various chronic diseases and mobility limitations,

often with concurrent mental and cognitive disorders

(Boulos2009);

 The typical expectations of elderly people

to have an independent and active life at their homes

or within their communities. The distance between

home and the care office center is a critical issue for

elderly people who choose for home care.

These people represent unique medical cases, as

subjects with various individual needs that should be

managed; the special focus is on the highest

important challenges of diseases prevention and

lifestyle management to timely clinical care and

follow ups.

The European Comission has stated 3 directions

of action (EC, 2007):

a) well ageing at working place or extending the

working activities;

b) well ageing within the community;

c) well ageing at home.

Within this general framework there are ongoing

concerns to find out efficient solutions supporting

medical care and continuous assistance for elderly

individuals at their home, with costs reduction too.

During the last few years a lot of real advances

was achieved in the home assistance electronic

services implementation; these services are

supported through the exponential development of

ICT technologies, enabling the improvement of the

elderly individuals life.

The home integrated tele-assistance systems

have an important position in this direction.

The tele-assistance integrated services represent

all the management, technical and economic

processes supporting and providing the assistance

services for people, especially at their homes; these

activities are performed using a platform containing

various networked devices, together with custom

software applications (Pușcoci, 2012).

A citizen-oriented services paradigm which

responds to his/her real needs becomes the central

pillar for the tele-assistance integrated services

development in order to meet several target

objectives, as shown in Figure 1 (Pușcoci, 2012).

The tele-assistance integrated services

149

Soviany S. and Puscoci S..

Multimodal Analysis for Behavioural Recognition in Tele-assistance Applications.

DOI: 10.5220/0005485901490154

In Proceedings of the 1st International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AgeingWell-

2015), pages 149-154

ISBN: 978-989-758-102-1

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

Figure 1: The tele-assistance integrated services target

objectives.

development should be done within a framework

containing the following main elements, according

to (Reem, 2012):

 Monitoring Services: remote measurement

and monitoring services. These services are used to

send real-time physiological information about

the patient’s condition through communications

(wired or wirelesss) lines and even over Internet.

Finally these monitoring services allow physicians

to adjust therapy to meet the patient’s changing

needs.

 Activity recognition : the psychophysical

skills identification for patients or elderly people in

order to optimally adjust the therapy actions.

 Behavior detection: the measurement and

detection of behavioral changes in patients’ typical

profile.

The assisted person's behaviour recognition

represents a novel decision support for the tele-

assistance systems. It requires for a lot of software

and hardware developments in order to implement

reliable solutions with optimal accuracy vs. cost-

effectiveness trade-off.

The papers proposes an approach for behavioural

recognition in which the individual conditions are

recognized using a multimodal analysis method.

This approach is an extension of our previous

work for biometric data. In this case the target

application is the accurate recognition of the human

behaviour in smart home environments, with main

focus on the home tele-assistance integrated services

of elderly people.

2 RELATED WORKS ABOUT

ACTIVITY RECOGNITION

The tele-assistance integrated services should be

implemented using ICT technologies, within a smart

environment/smart home which allows to develop

home applications. This approach aims the assisted

persons to perform their daily activities and to

benefit from a set of telemonitoring and emergency

actions services.

The tele-assistance integrated services

implementation is a challenging task because the

only remote monitoring of some medical or

environmental parameters does not efficiently

approach the home elderly people care and

assistance.

The related works about telemonitoring, tele-

assistance and ambient assisted living applications

include a lot of published solutions and methods for

home people daily activities recognition and human

behaviour recognition.

In (Rodrigo, 2009) the authors developed a

feature selection method for Human Activity

Recognition. They used a large feature set as

descriptors for the human activities and also applied

Best First Search and Genetic Algorithms for the

optimal feature subset selection to maximize the

accuracy of a Hidden Markov Model generated from

those features. The approach was compared against

other published techniques for human activities

classification using video sensors. The optimal

feature selection is justified by the fact that sensor

data are typically noisy; the best attributes should be

retrieved before the classifier traning. All the

features were generated from image sequences.

In (Young-Seol Lee, 2011) a 3D accelerometers-

based activity recognition method was proposed.

The authors developed an activity recognition

system for smartphones in which the uncertain time-

series acceleration signal was processed using

hierarchical Hidden Markov Models. The model was

designed in respect to the typical resources

constraints of the mobile devices (memory storage

and computational power). The overall proposed

hierarchical probabilistic model for humans

activitivies recognition combined 2 different

probabilistic models, a continuous HMM and a

discrete HMM. This hierarchical approach for

probabilistic models seems to be more reliable if the

patterns could be divided into smaller units. The data

acceleration from a 3-axis accelerometer on a

smartphone is initially transferred to a low-level

HMM for human actions classification. Then a high-

level HMM is applied to identify the human actions.

Healthcare insurance,

through telemonitoring,

for people with chronic

diseases

To integrate

healthcare and

social care services

To meet the elderly

people demands for

living at their own home

OBJECTIVES

Health

Services

Social

Services

Social+Health

Services

tele-assistance

integrated

services

To improve the life

quality for

individuals and their

families

To provide an

independent life style

for people elderly

To adopt solutions

supporting the elderly

people acceptability to use

electronic technology at

home in daily life

ICT4AgeingWell2015-InternationalConferenceonInformationandCommunicationTechnologiesforAgeingWelland

e-Health

150

In (Ugur Toreyin, 2008) the authors approached

the falling person detection. A safe and active life

for elderly people could be supported in a home

environment with sound, passive infrared (PIR) and

vibration sensors. In their approach the falls

detection is based on the simultaneous analysis of

signals produced by sound, PIR, and vibration

sensors. The classification is done by training

Hidden Markov models (HMM) for regular and

abnormal actions of elderly people. The final

decision resulted from the fusion of HMMs

decisions. The classifier was trained according to the

possible human states.

(Zouba, 2009) defined a method for home elderly

people activity recognition using a multisensor

approach with video cameras and environmental

sensors. The authors applied a high-level (event)

fusion with a combination of video and environment

events. They used heterogeneous sensor data for the

home elderly people activities recognition. This task

is performed with a data fusion method, according to

the application requirements.

In (Oliver Brdiczka, 2008), the authors

approached learning and recognition of human

behavioral patterns given multimodal data from a

smart home environment. The proposed method

general goal was to achieve a high-level contextual

model for human behavior. The work was mainly

related by the problem of automatically human

behavior recognition in a smart home environment.

The human behavioral patterns were learned and

recognized using a multimodal data processing

approach for video and audio information in a smart

home.

In (Rim, 2012) the authors focused on a major

challenge of context-aware computing and

intelligent environments: the acquisition and

modelling of heterogeneous context data. The key

issues are the various granularity degrees for the

human activities. The authors considered that

ontology-based activity models are able to support

interoperable multilevel activity recognition. They

applied probabilistic description logics (DLs) for

multilevel activities identification in smart

environments.

3 MULTIMODAL METHOD FOR

THE BEHAVIOURS

RECOGNITION

In our opinion the behavioural analysis of the home-

assisted person is a suitable way to get meaningful

information about his/her state. It allows to develop

prevention programmes for risk situations; these

should enable medical and social actions for early

solving of the detected problems.

In our approach the designed system includes

one or several sensors networks which are deployed

at the assisted person home. The resulting sensor

data are provided as input set for a multimodal

processing and analysis module. The further

processed data are used then to generate a specific

behavioural pattern for each end-user.

The generated model is learned by the overall

tele-assistance system. The further matching against

the current test sample allows to early detect the

behavioural anomalies. These detected outliers are

then reported to the surveillance unit (or tele-

assistance dispatcher) that should be responsible for

the required action in each case.

The specific behavioural model results from the

various environmental and medical sensors data

analysis. The achieved information should have

enough relevance in order to be useful for finding

out the influence of health state and living

environment on the individual behaviour.

The proposed method for behaviour recognition

is actually an extension of our recent works on

multimodal biometric systems with hierarchical

classifiers design. These biometric developments

were focused on security applications for access

control (Soviany, 2012), (Soviany, 2013), (Soviany,

2014). Now we extend the hierarchical classification

model from the previous biometric recognition to

another application concerning the human behaviour

recognition.

We think about this extension actually as a kind

of behavioural biometric, but with a different target

instead of the typical security applications. This time

the new target is the tele-assistance in smart homes

environments.

Within this proposed framework we consider that

the multimodal analysis is a reliable tool allowing to

efficiently exploit and correlate various data

concerning the individuals and their position, health

state and behaviour. In this approach several

different information sources could be correlated in

order to accurately identify the true person’s state.

The general architecture for the proposed system

is depicted in figure 2 (with focus on the processing

stages and decisions generation).

The multimodal analysis method for behavioural

data processing is sequenced into the following basic

operations:

 Behavioural data types definition;

 Thresholding;

MultimodalAnalysisforBehaviouralRecognitioninTele-assistanceApplications

151

Figure 2: The general architecture for the behavioural

patterns multimodal processing system.

 Dimensionality optimization;

 Matching;

 Multimodal fusion

3.1 Behavioural Data Types Definition

The behavioural data types definition is essential for

the further individuals behavioural patterns

generation.

A behavioural pattern for one individual subject

of tele-monitoring is a set of values for certain

application-specific variables, attributes or

parameters; these variables are chosen just within

the design stage such as to provide the best

relevance for the person attitudes, movements or

postures.

In the proposed approach for each person

condition, state or posture we define a separate class

of behavioural patterns; this definition is done

according with the application specific requirements.

The sources of behavioural data are various

sensors. These sensors are typically linked in

wireless sensors networks with hierarchical

architectures, depending on the tele-monitoring

application complexity. The input data quality and

behavioural patterns generation accuracy are critical

factors for the overall system performance.

3.2 Thresholding

A critical issue for the suitable correlation between

the individual state/posture/condition and the

behavioural patterns is the relevance degree of the

chosen attributes (variables, parameters). These

variables are used as feature sets describing the

attitudes, postures and/or other current conditions of

the subject.

A detailed analysis of the behavioural features

and patterns is required in order to define a set of

suitable thresholds for each behavioural attribute.

This thresholding supports the correlation between

the person condition and his/her behavioural pattern;

this could be done with suitable thresholds for each

of the chosen attributes.

This thresholding-based approach provides an

accurate and computational-effective classification

of the current state for the monitored (assisted)

subject. The classifier outputs support the decision-

making step.

The main drawback of the thresholding-based

approach is its sensitiveness in various practical

applications with different specific requirements.

This is especially related by choosing the suitable

thresholds for the various behavioural patterns and

for their components, respectively.

On the other hand the influence of the

environmental parameters on the certain individual

conditions should be considered for the thresholds

selection.

This is why we should also consider other

additional options to provide the behavioural

features relevance but only after a serious

assessment of the available data according to the real

application requirements.

3.3 Dimensionality Adjustment

The achieved datasets contain several values for

various parameters. These values are typically

grouped into certain behavioural patterns

representing datapoints (or feature vectors) within a

multi-dimensional space.

The resulted higher dimensionality is a

significant issue for the behavioural patterns

classification. A high-dimensional feature space

typically requires for a lot of training samples per

class to design an accurate classifier; the multi-

dimensional space should be covered with many

training datapoints.

On the other hand a high dimensional feature

space for the behavioural patterns is not always very

useful to provide the best accuracy for behaviours

ICT4AgeingWell2015-InternationalConferenceonInformationandCommunicationTechnologiesforAgeingWelland

e-Health

152

recognition; this is because certain attributes could

have not the same relevance for the human

behaviour.

Therefore the feature selection procedure is often

required to optimally adjust the resulted datasets of

behavioural patterns. For this step we intend to test

many of the actual approaches in feature selection,

either optimal (exhaustive) or suboptimal (non-

exhaustive) techniques for feature selection. We will

look for the best trade-off execution time vs. features

optimality.

3.4 Matching

The further processing step of the behavioural

patterns recognition performs their matching against

the reference data. During the ongoing research we

are still exploring for several design solutions,

models and algorithms that should be used for the

matching operation.

This operation is actually very challenging

because it exhibits the highest computational

complexity degree, which is mainly given by the

high-dimensional feature space. This is true despite

of the previous feature selection step. Another

critical issue results from the intrinsic dynamic

feature of the human behaviour; in this case the

reliable pattern identification still remains a difficult

task. In most of the actual related works HMM-

based models were applied for human activity

recognition with optimal results just for their

applications.

However we intend to extend our works in

biometric data hierarchical classification and

similarity scores computation to human behaviour

recognition. Therefore we consider the following

design solutions for the matching operation:

 The distance-based approach in which a

similarity ranking is directly evaluated in the

multi-dimensional feature space of the

behavioural patterns, just in a similar way as

for the large-scale available biometric systems.

We should apply various distance measures,

for instance Mahalanobis distance; this is

useful when it is important to optimally exploit

the behavioural feature correlation as

proceeded for typical biometric data. This

approach is strongly dependent on the features

thresholding and also exhibits a significant

computational complexity;

 the supervised or semi-supervised learning-

based approach in which a hierarchical

classifier is designed and suitable trained with

the available behavioural patterns. The model

will classify the behavioural data in several

stages, according to the classes importance. For

the most critical behavioural states of an

individual we will firstly apply a detection

stage; a detector is a classifier which is only

trained for one target class (Soviany, 2014).

We previously applied this classification

design solution for biometric data and the

achievements proved a significant

improvement in identification accuracy; this

improvement was provided by the detectors

design principle with one class classification

approach, saving a lot of computational

expenses. On the other hand there are

additional issues to be considered in this case,

such as the applied cross-validation procedure

which is very often required to overcome

certain problems of many classification

systems (like overfitting).

3.5 Multimodal Fusion

The final decision concerning the real condition of

the individual should result from a combination of

scores or even separate decisions issued from the

various behavioural pattern previously processed.

Actually during the ongoing research we will

evaluate several score or decision-level fusion rules.

We will consider the typical rules which are already

applied in most of the actual multimodal biometric

systems, such as sum rule, mean rule, minimum

score rule or maximum score rule; for the sum and

mean rules we also consider their weighted versions.

The main challenge of this step is that not all the

behavioural patterns have the same critical

significance in showing the worst case of the

subjects. This is why sometimes a hierarchical

decision structure should be applied, while

considering several priority levels for the various

behavioural patterns.

4 CONCLUSIONS

We propose a multimodal approach for human

behaviour recognition in which we extend our

previous developments for biometric data. The

behavioural recognition is performed with a

hierarchical optimized classifier in which several

decision stages are followed to accurately detect and

recognize the real state of the assisted person.

The proposed approach allows to efficiently deal

with various behavioural patterns using a multi-stage

classifier, also including a one-class classification

MultimodalAnalysisforBehaviouralRecognitioninTele-assistanceApplications

153

stage (detector).

Another difference between our proposal and the

actual systems for behavioural recognition in smart

environments is that we apply the hierarchical

classifier on multiple data types, not only on images

and sounds. This will be a real challenge for the

ongoing research.

ACKNOWLEDGEMENTS

This paper was financed through INSCC grant no.

0412/2015 with Ministry of Education and Scientific

Research

REFERENCES

EC 2012 Ageing Report. https://ec.europa.eu/digital-

agenda/en/news/2012-ageing report-economic-and-

budgetary-projections-27-eu member-states-2010-

2060.

Boulos, MK, Castellot Lou, R, Anastasiou, et al.,2009

Connectivity for Healthcare and Well-Being

Management: Examples from Six European Projects,

Int J Environ Res Public Health. 2009 July; 6(7):

1947–1971.

EC 2007 “Ageing well in the Information Society”, COM

(2007) 332final, Bruxelles. http://www.capsil.

org/files/Action%20Plan%20on%20.Information%20a

nd%20Communication%20Technologi %20 and% 20

Ageing.pdf.

Pușcoci, Sorin, 2012. Tele-assistance integrated services,

In Telecommunications, Anul LV, nr. 2.

Reem Al-Attas, Abdulsalam Yassine, Shervin

Shirmohammadi, 2012. Tele-medical applications in

home-based health care. In 2012 IEEE International

Conference on Multimedia and Expo Workshops.

Soviany, Sorin, Puşcoci, Sorin, 2014 An Optimized

Multimodal Biometric System with Hierachical

Classifiers and Reduced Features. In IEEE

International Symposium on Medical Measurements

and Applications (MeMeA),

Soviany, Sorin, Puşcoci, Sorin, 2013. A Feature

Correlation-based Fusion Method for Fingerprint and

Palmprint Identification Systems, In The 4th IEEE

International Conference on E-Health and

Bioengineering - EHB 2013 Grigore T Popa

University of Medicine and Pharmacy, Ia§i, Romania,

Soviany, Sorin, Puşcoci, Sorin, Mariana Jurian, 2012 A

multi-level hierarchical biometric fusion model for

medical applications security, In the 8th Annual

International Conference on Computer Science and

Information Systems (INFOS2012), Atena, Grecia,

Rodrigo Cilla, Miguel A. Patricio, Jesus Garcıa, Antonio

Berlanga and Jose M. Molina, 2009 Recognizing

Human Activities from Sensors Using Hidden Markov

Models Constructed by Feature Selection Techniques,

In Algorithms 2009, 2, 282-300;

oi:10.3390/a2010282.

Young-Seol Lee and Sung-Bae Cho 2011, Activity

Recognition Using Hierarchical Hidden Markov

Models on a Smartphone with 3D Accelerometer, In

E. Corchado, M. Kurzyński, M. Woźniak (Eds.): HAIS

2011, Part I, LNAI 6678, pp. 460–467, 2011.

B. Ugur Toreyin, E. Birey Soyer, Ibrahim Onaran, and A.

Enis Cetin, 2008, Falling Person Detection

UsingMultisensor Signal Processing, In Journal on

Advances in Signal Processing Volume 2008, Article

ID 149304,

Nadia Zouba, Francois Bremond, Monique Thonnat. 2009,

Multisensor Fusion for Monitoring Elderly Activities

at Home. 6th IEEE International Conference on

Advanced Video and Signal Based Surveillance

AVSS09, Sep 2009, Genoa, Italy.

Oliver Brdiczka, Matthieu Langet, Jérôme Maisonnasse,

and James L. Crowley 2008, Detecting Human

Behavior Models From Multimodal Observation in a

Smart Home In IEEE Transactions on automation

science and engineering, 2008.

Rim Helaoui, Daniele Riboni, Mathias Niepert, Claudio

Bettini, Heiner Stuckenschmidt, 2012, Towards

Activity Recognition Using Probabilistic Description

Logics, In Activity Context Representation:

Techniques and Languages AAAI Technical Report

WS-12-05.

ICT4AgeingWell2015-InternationalConferenceonInformationandCommunicationTechnologiesforAgeingWelland

e-Health

154