A Cognitive-IoE Approach to Ambient-intelligent Smart Home

Gopal Jamnal and Xiaodong Liu

School of Computing, Edinburgh Napier University, Edinburgh, U.K.

Keywords: Intelligent Inhabited Environment, Ambient Intelligent Smart Home, Activity Pattern Recognition, Cognitive

IoTs, and Cyber Physical System.

Abstract: In today’s world, we are living in busy metropolitan cities and want our homes to be ambient intelligent

enough towards our cognitive requirements for assisted living in smart space environment and an excellent

smart home control system should not rely on the users' instructions. Cognitive IoE is a new state-of-art

computing paradigm for interconnecting and controlling network objects in context-aware perception-action

cycle for our cognitive needs. The interconnected objects (sensors, RFID, network objects etc.) behave as

agents to learn, think and adapt situations according to dynamic contextual environment with no or minimum

human intervention. One most important recent research problem is “how to recognize inhabitant activity

patterns from the observed sensors data”. In this paper, we proposed a two level classification model named

as ACM (Ambient Cognition Model) for inhabitant’s activities pattern recognition, using Hidden Markov

Model based probabilistic model and subtractive clustering classification method. While subtractive

clustering separates similar activity states from non-similar activity state, a HMM works as the top layer to

train systems for temporal-sequential activities to learn and predict inhabitant activity pattern proactively. The

proposed ACM framework play, a significant role to identify user activity intention in more proactive manner

such as routine, location, social activity intentions in smart home scenario. The experimental results have

been performed on Matlab simulation to evaluate the efficiency and accuracy of proposed ACM model.

1 INTRODUCTION

In recent years, IoT has envisioned the hardware

technologies that let mobile and embedded devices to

better exploit the web-internet features to connect

anytime, anyplace, which enhanced interactive

experience of people centric Cognitive Internet of

Things (Feng et al. 2017). As earlier claimed by

Satyanarayanan (2001), great technology inventions

are those, that dissolve themselves into everyday life

and be invisible for human consciousness. As a result,

such research visions are making futuristic scenarios

of Ambient Intelligence smart environments more

promising into the reality of our everyday lives

activities. The Cognitive IoT, overlaps the various

research areas of pervasive computing, wireless-

sensor networking, IOTs, artificial intelligence,

machine learning and context-aware computing and

cyber physical system. In CIoT, smart spaces extend

the functionality of ambient intelligence toward more

proactive possibilities, where smart environment not

only monitors people for tasks, or support them by

executing their requests, but also influences and

changes their plans and intentions. Also by EU report,

pervasive computing will be the next wave of new

ICT innovation in the next five years, and it’s said by

2020 it will be one major type of ICT system.

Therefore, CIoT has been viewed from the industry

and the academic world as a main pillar of an

upcoming industry revolution. The rapid use of

interconnected network devices in healthcare

systems, smart vehicles, transportation, classrooms,

production units, smart homes, agriculture, would

result a technological revolution in ubiquitous

connectivity, computing and communication. (Ricci

et al. 2015 and Perera et al. 2014 and Feng et al. 2017)

In this paper our focus is on Cognitive IoT application

in smart home for automated recognition of

inhabitant activities to enhance the independent living

experience to improve daily quality of life. In smart

home scenario, obtained sensor data of inhabitant’s

activities interactions sequences within the

environment is segmented and can be labeled as

specific activity instance with description. The

detected activity instances are used to train an activity

recognition model as a result, the trained model are

302

Jamnal, G. and Liu, X.

A Cognitive-IoE Approach to Ambient-intelligent Smart Home.

DOI: 10.5220/0006304103020308

In Proceedings of the 2nd International Conference on Internet of Things, Big Data and Security (IoTBDS 2017), pages 302-308

ISBN: 978-989-758-245-5

then used to classify and assign a label to new activity

instance. As Nazerfard et al. (2010) argued that

discovering the order of activities can be effectively

used for predicting the next activity in a home

automation system using their temporal relation

information. The activities pattern can be recognized

by using machine-learning techniques such as HMM,

Navies Bayes, Decision Tree, ANN and SVM and

KNN etc. (Fahad et al. 2014 and Somov et al. 2013

and Wu et al. 2014)

The main aim of CIoT in smart home environment is

to improve quality of life via developing a ambient

intelligent living environment. The communication

layer of multiple sensors can control home appliance

via actuator/device controller to help inhabitant into

the daily activities. In other words, CIoT works as

brain, where raw data gathered from sensors and

information collected and fused into decision making

unit, for computing controlling commands to achieve

specific goals. In such cases every home appliance

can be programmed according to inhabitant need and

living patterns. In industry, we have few smart

solutions such as smart grids, electric meters, security

controls system, lighting system, which can be

programmed to customized as per individual desire

(Feng et al.2017).

The research project aims to develop a novel dynamic

architecture including its related models and

mechanism to Cognitive-IoE based smart homes,

where the functionality of ambient intelligence is

extended towards more proactive possibilities, i.e.,

the smart environment not only monitors

people/devices for tasks, or support them by

executing their requests, but also influence and

change their plans and intentions. In the dynamic

environment, a home is equipped with multiple

sensor (motion, light, noise) to perceive the

environmental data in consistent/inconsistent state

and preprocessed for further activity (similar/non-

similar) classification. Human activity sequences can

be analyzed from sensor data using their temporal

values and transferred to an inference engine to

recognized their daily activity patterns as routine,

location and social contexts.

The rest of this paper is organized as follows. Section

2 summaries related work on activity recognition

done in the community. A bottom up approach to

inhabitant activity recognition in smart space is

presented as the ACM architecture Section 3. The

various classification based evaluation methods are

presented in Section 4. The work is concluded in

Section 5 and future work is discussed in Section 6.

2 SMART HOME: INHABITANT

ACTIVITY RECOGNITION

PROJECTS

In recent years, learning and understanding the

observed activity and event mining are the central

research area to smart home studies. Activity refers to

complex behaviors consisting of a sequence of action

and overlapped action that can be performed by a

single individual or several individuals interacting

with each other. Some significant smart home activity

recognition research work has been done in Care-lab,

CASAS, Grator-Tech HIS, Aware Home, iDorm, and

MavHom projects. In particular, the process of

activity recognition can be divided in four steps such

as i) sensing, ii) data-preprocessing, iii) data

modelling Feature extraction and iv) feature

selection. The major research work is in progress by

the tech giant IBM Watson, where cognitive

appliances talk to each other and the central

computing unit works as personalized digital assistant

for granting access and controlling various appliances

example e.g. smart locks, digital reminders, etc. IBM

Watson’s cognitive IoT vision is to create a custom

tailored environment for individual residents by

adapting their preference and patterns, which not only

ensure better security, predictive maintenance tasks

and alert system but also saves time and money of

individuals, working as personal assistants (IBM

Watson, 2017).

2.1 Subtractive Clustering: Pattern

Recognition In Smart Home

While we talk about ambient intelligent smart home,

the need of clustering methods arises to find

relationships between observed datasets. As we

know, clustering based classification is a well-known

approach to extract knowledge from obtained datasets

by dividing datasets into discreet classified clusters.

Two major clustering methods, K-means and the

subtractive clustering can resolve the problem of

separating similar and non-similar activities from

given datasets (sensor’s data). The k-means clustering

algorithms works on pre-segmented (known) clusters

numbers, where we assume the number of clusters in

advance for given datasets. Sometimes, it is not the

case to have prior knowledge about the required

A Cognitive-IoE Approach to Ambient-intelligent Smart Home

303

number of clusters for obtained data sets. Also it

makes system less flexible to identify appropriate

cluster due to limited number of clusters. On the other

hand, to improve the learning ability we can take the

help of subtractive clustering algorithms to find

patterns in inhabitant behavior. As figure 1 shows,

subtractive clustering methods estimate the cluster

center and select the data point with highest potential

value to be the first cluster center, and later remove

all data points within the vicinity of the first cluster

center in order to determine the next data cluster and

its center location. The process keeps going on until

all the remaining datasets identify their radius of a

cluster center. Sub-clustering is a quick one pass

algorithm, moreover it works in the context that the

expert has no idea about the cluster number and

specifications. In the same way, the smart home

control system depends on the simulation of human

experience to make itself an intelligent control system

to perceptually make judgement to the external

environment variables, which are very strong,

moderate, and very weak (Amirjavid et al. 2014 and

Fahad et al. 2014).

Figure 1: Subtractive clustering for data pre-processing.

2.2 Hidden Markov Model: Pattern

Recognition In Smart Home

HMM is the most generative temporal probabilistic

model, It is applied to find hidden states (y1, y2, . . . ,

yT), in the observed time series data sequence (x1, x2,

. . . , xT). HMM is fundamentally based on the

independence of hidden states and the condition

independence of observation parameters stipulating

that p(xt|yt, x1, x2 . . . xt- 1, y1, y2 . . . yt－1) = p(xt|yt).

The observable state at time t, xt, depends only on the

current hidden state yt. That is, the probability of

observing xt while being at yt is independent of all

other observable and hidden variables. The joint

probability p(x, y) of the observations and hidden

states can be factorized as follows:

The links between the hidden states are labeled with

the transition probabilities and those between the

hidden states and the observed sequence are labeled

with the emission probabilities. Using the initial and

transition probability matrix of the observable

Markov model, it is possible to calculate the

probability of any new activity sequences from the

current activity sequence. That is the model can

estimate the probabilities of new sequence, as shown

in figure 2, HMM can solve three problems such as:

Evaluation, Decoding and Learning problem. The

evaluation problem uses a forward-backward

algorithm to evaluate the probability of efficient

computation based on a given set of observations. The

learning problem optimizes parameters of a model to

better describe the observations using the

expectation-maximization (EM) algorithm. Finally,

the decoding problem uses the Viterbi algorithm to

find the most likely state sequence given an

observation sequences. (Benmansour et al. 2016)

Figure 2: Problem solving stages in HMM. (Benmansour et

al. 2016).

As HMM jobs to identify the most likelihood

sequences in given observed data vectors, so it is a bit

challenging to adjust or re-estimate the model

parameters to obtain the maximized probability of

likelihood sequence vector. Therefore, an iterative

approach of Baum-welch algorithm can be applied to

re-estimate the HMM parameters sub-optimally, as

expectation maximization(EM) method. Steps are :

(1) λ

= initial model

(2) compute new λ based on 𝜆

and given observation

sequence 𝑥⃗

(3) if log Pr(𝑥⃗/𝑦) – log Pr(𝑥⃗/𝜆

) < Γ stops, else set

𝜆

= λ and repeat step (2)

Here, Γ represents the minimum tolerance value

between two subsequent model.

IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security

304

Afterwards, the forward-backward algorithm places

the re-estimated parameter in set of equations as

below;

Here, Q=states sequences (q1,q2, ..., qk)

λ = HMM model

𝑋

→ = input data vector<x1,x2,x3…, xk> of

observation sequence.

(Hassan et al. 2013)

3 A BOTTOM UP APPROACH:

ACM (AMBIENT COGNITION

MODEL) FOR INHABITANT

ACTIVITY RECOGNITION

In particular, when we want to apply machine

learning algorithms for activity pattern recognition in

smart home scenario, we follow the sequential

occurrence of regular activity in order to find changes

in patterns of individual lifestyle. In addition, there

are many methods for activity recognition in the

related area, therefore it become important to

consider general classification and examination of

each approach and their implementation constraints

for specific problem solution (Zolfaghari and

Keyvanpour, 2016). In our research work, we applied

the bottom-up approach for human activity

recognition based on the data-driven probabilistic

model. In contrast, the key contribution of our model

is to identify user’s key intentions based on observed

sensors data. We used, subtractive clustering and

Hidden Markov Model combination to recognize

activities patterns in smart homes. As we applied the

two level classification model in our proposed ACM

framework in figure 3, inhabitant activity recognition

is done in four stages; sensing, pre-processing, data

modelling and context relevancy check such as social,

environmental and activity intentions.

In the first step, the sensor raw data are collected,

containing information about objects, which have

temporal information of each activity event.

Furthermore, in order to separates the relationship

between similar and non-similar activities, a

subtractive clustering methods is applied to find

relevant clusters in datasets. The sub-clustering

algorithm works on radius parameter to identify the

number of clusters. As a result, the large set of data

problem is subdivided into the small datasets

problem. Afterwards, using initial transition

Figure 3: Proposed ACM (Ambient-Cognition Model)

framework components.

probability and emission matrix of the observed

sequences, we applied Baum-welch algorithm to re-

estimate the HMM probability (transition and

emission) matrix input parameters. later the ACM

trained for 111 datasets using a re-estimated

probability matrix.

We applied Viterbi algorithm to compute most

likelihood sequence (probable-path) to find the

maximum occurrence over all possible state

sequences of given observed sequence datasets,

which make our ACM model more ambient

intelligent and proactive to understand inhabitant

intentions. Furthermore, likelihood sequence is

evaluated comparing to real time observed date

sequence to check system accuracy and intelligence

to identify recognized patterns. The following Viterbi

equation can find most likely sequences of hidden

states of given observations:

Furthermore, we identify the user’s intentions from

HMM likelihood sequences, we can subdivide them

into further three categories such as activity intention

of users’ movements for specific work, and location

intention of user’s specific movement and appearance

on specific locations. In particular, a routine intention

to identify user’s daily activities can be outlined with

repetitive task performed. Along with this, the

activity knowledge base will be storing intentions for

historical information purpose to link up for future

reference.

A Cognitive-IoE Approach to Ambient-intelligent Smart Home

305

4 EVALUATION METHODS FOR

ACM FRAMEWORK

In an ambient environment, ADL (Activity daily

living) can be evaluated in many different ways such

as training and testing of each routine activity.

Besides, there might be possibility of data scarcity of

each activity, which might create problem of

accuracy and performance evaluations. Above all,

major problem might occur when inhabitant is

reluctant to reveal their behavioral data due to privacy

and ethical consideration. Therefore, a researcher

should consider the availability of activities data sets

before analyzing human activity recognition in

ambient intelligent smart spaces.

For our research, we used existing ADL (activity

daily living) datasets of multiple ubiquitous sensors

placed in various location provided by MIT media

Lab from Tapia (2003) research work. We will be

using provided activity sequence of 111 samples of

single user activity which are labeled based on each

activity type.

Table 1: Activity daily living with their unique ID.

Activity Daily Living

Preparing dinner

Watching TV

Toileting

Preparing breakfast

Preparing lunch

Dressing

Taking medication

bathing

talking on phone

cleaning

However, the main problem in MIT media lab data

set is that, no time interval has been set in

experimental data sets. As a result, number of activity

states are not equal for each day. For example, some

days have more activities sensors values compare to

other days (day1 captured 5 activities, day2 have 9

activities captured). It could have been better if value

0 is provided where no activities event happened for

specific day to maintain number of activity states in

correct sequence manner. Hence, all days would have

equal set of data value in matrix. In other words,

every row has the same length. The below figure 4,

represents the 10 activities (mentioned in table 1) of

111 observed sequences from MIT media lab during

19/4/2003 to 31/4/2003.

Figure 4: MIT media lab activity datasets.

Furthermore, these 10 household activities are sub-

clustered into 3 main activities clusters such as

personal needs (1), domestic work (2) and relaxing

(3) labels, mentioned in below table 2.

Table 2: three-activity cluster of household.

Activity Cluster

Type

Personal needs(1)

Taking medication

bathing

talking on phone

Dressing

toileting

Domestic work(2)

Preparing lunch

Preparing dinner

cleaning

preparing breakfast

Relaxing(3)

Watching TV

Matlab 2016 version were used for test experiments,

furthermore 111 observed activity datasets have been

labeled into 3 main activities. Initial transition and

emission matrix has been defined based on prior

knowledge base. Afterwards, using Baum-welch

algorithm further re-estimation of transition and

emission probability matrix has been successfully

achieved. Correspondingly, Matlab train the system

for 111 observed sequences and Viterbi algorithm

identified most likelihood activity sequences with the

IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security

306

77% of accuracy, shown in figure 5 as Matlab

simulation result.

Figure 5: HMM likelihood activity sequence compares to

observed sequence.

5 CONCLUSION

This paper proposed a new type of ambient intelligent

architecture for smart homes to understand the

inhabitant intentions proactively. combining

knowledge from traditional context aware pervasive

systems with modern era Cognitive IoT technologies.

The building block of ACM architecture is well

equipped with artificial intelligence machine learning

algorithms of subtractive clustering and Hidden

Markov model to recognize patterns in daily activity

living. ACM ensure the comfortable living

environment for inhabitant as its primary goal.

Moreover, the ACM model can be successfully

applied in elderly health care systems, smart

classrooms, and smart spaces for automated

environment. While given MIT media lab data sets

itself a detailed activities observation of inhabitants

but could have been lot more better if have more data

samples are available to avoid the data scarcity

problem. In addition, the provided MIT media lab

datasets could have better attributes in well-organized

time set intervals. However, our ACM model well

trained over provided data samples and successfully

achieved 77% of likelihood sequence accuracy.

6 FUTURE WORK

As part of our ongoing work, we are extending our

ACM to adapt a fuzzy-rule based system for

executing tasks for physical world. We plan to

combine ACM with device controller by inferring

fuzzy rule activations. Thus, it will be an excellent

device controller for all appliances in smart

homes/spaces.

In addition, we would be creating our own smart

home lab, where sensors data will be collected in

predefined time set interval to maintain data accuracy

and uniformity. As a result, the data scarcity problem

would be avoided automatically and the result would

be simulated with increased pattern recognition

accuracy.

REFERENCES

Amirjavid, F., Bouzouane, A., & Bouchard, B. (2014). Data

driven modeling of the simultaneous activities in

ambient environments. Journal of Ambient

Intelligence and Humanized Computing, 5(5), 717-

740.

Benmansour, A., Bouchachia, A., & Feham, M. (2016).

Multioccupant activity recognition in pervasive smart

home environments. ACM Computing Surveys

(CSUR), 48(3), 34.

Fahad, L. G., Tahir, S. F., & Rajarajan, M. (2014). Activity

recognition in smart homes using clustering based

classification.

Feng, S., Setoodeh, P., & Haykin, S. (2017). Smart Home:

Cognitive Interactive People-Centric Internet of

Things. IEEE Communications Magazine, 55(2), 34-

39.

Hassan, M. R., Ramamohanarao, K., Kamruzzaman, J.,

Rahman, M., & Hossain, M. M. (2013). A HMM-based

adaptive fuzzy inference system for stock market

forecasting. Neurocomputing, 104, 10-25.

IBM Watson, (2017) Watson Internet of Things [Online]

https://www.ibm.com/internet-of-things/iot-zones/iot-

buildings/cognitive-systems-in-buildings/ [accessed

14/01/2017].

MIT (2003) Using Machine Learning for Real-time

Activity Recognition and Estimation of Energy

Expenditure [Online] <http://alumni.media.mit.edu

/~emunguia/pdf/PhDThesisMunguiaTapia08.pdf>

[Accessed 12/01/2017].

Munguia Tapia, E. (2003). Activity recognition in the

home setting using simple and ubiquitous

sensors (Doctoral dissertation, Massachusetts Institute

of Technology).

Nazerfard, E., Rashidi, P., & Cook, D. J. (2010, December).

Discovering temporal features and relations of activity

patterns. In 2010 IEEE International Conference on

Data Mining Workshops (pp. 1069-1075). IEEE.

Perera, C., Liu, C. H., Jayawardena, S., & Chen, M. (2014).

A survey on internet of things from industrial market

perspective. IEEE Access, 2, 1660-1679.

Ricci, A., Piunti, M., Tummolini, L., & Castelfranchi, C.

(2015). The Mirror World: Preparing for Mixed-Reality

Living. IEEE Pervasive Computing, (2), 60-63.

A Cognitive-IoE Approach to Ambient-intelligent Smart Home

307

Satyanarayanan, M. (2001). Pervasive computing: Vision

and challenges.Personal Communications, IEEE, 8(4),

10-17.

Somov, A., Dupont, C., & Giaffreda, R. (2013). Supporting

Smart-city Mobility with Cognitive Internet of Things,

1–10.

Wu, Q., Ding, G., Xu, Y., Feng, S., Du, Z., Wang, J., &

Long, K. (2014). Cognitive Internet of Things: A New

Paradigm beyond Connection. IEEE Internet of Things

Journal, PP(99), 1–1. http://doi.org/10.1109/JIOT

.2014.2311513.

Zolfaghari, S., & Keyvanpour, M. R. (2016). SARF: Smart

activity recognition framework in Ambient Assisted

Living. In Computer Science and Information Systems

(FedCSIS), 2016 Federated Conference on (pp. 1435-

1443). IEEE.

IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security

308