A Cognitive-IoE Approach to Ambient-intelligent Smart Home
Gopal Jamnal and Xiaodong Liu
School of Computing, Edinburgh Napier University, Edinburgh, U.K.
Keywords: Intelligent Inhabited Environment, Ambient Intelligent Smart Home, Activity Pattern Recognition, Cognitive
IoTs, and Cyber Physical System.
Abstract: In today’s world, we are living in busy metropolitan cities and want our homes to be ambient intelligent
enough towards our cognitive requirements for assisted living in smart space environment and an excellent
smart home control system should not rely on the users' instructions. Cognitive IoE is a new state-of-art
computing paradigm for interconnecting and controlling network objects in context-aware perception-action
cycle for our cognitive needs. The interconnected objects (sensors, RFID, network objects etc.) behave as
agents to learn, think and adapt situations according to dynamic contextual environment with no or minimum
human intervention. One most important recent research problem is “how to recognize inhabitant activity
patterns from the observed sensors data”. In this paper, we proposed a two level classification model named
as ACM (Ambient Cognition Model) for inhabitant’s activities pattern recognition, using Hidden Markov
Model based probabilistic model and subtractive clustering classification method. While subtractive
clustering separates similar activity states from non-similar activity state, a HMM works as the top layer to
train systems for temporal-sequential activities to learn and predict inhabitant activity pattern proactively. The
proposed ACM framework play, a significant role to identify user activity intention in more proactive manner
such as routine, location, social activity intentions in smart home scenario. The experimental results have
been performed on Matlab simulation to evaluate the efficiency and accuracy of proposed ACM model.
In recent years, IoT has envisioned the hardware
technologies that let mobile and embedded devices to
better exploit the web-internet features to connect
anytime, anyplace, which enhanced interactive
experience of people centric Cognitive Internet of
Things (Feng et al. 2017). As earlier claimed by
Satyanarayanan (2001), great technology inventions
are those, that dissolve themselves into everyday life
and be invisible for human consciousness. As a result,
such research visions are making futuristic scenarios
of Ambient Intelligence smart environments more
promising into the reality of our everyday lives
activities. The Cognitive IoT, overlaps the various
research areas of pervasive computing, wireless-
sensor networking, IOTs, artificial intelligence,
machine learning and context-aware computing and
cyber physical system. In CIoT, smart spaces extend
the functionality of ambient intelligence toward more
proactive possibilities, where smart environment not
only monitors people for tasks, or support them by
executing their requests, but also influences and
changes their plans and intentions. Also by EU report,
pervasive computing will be the next wave of new
ICT innovation in the next five years, and its said by
2020 it will be one major type of ICT system.
Therefore, CIoT has been viewed from the industry
and the academic world as a main pillar of an
upcoming industry revolution. The rapid use of
interconnected network devices in healthcare
systems, smart vehicles, transportation, classrooms,
production units, smart homes, agriculture, would
result a technological revolution in ubiquitous
connectivity, computing and communication. (Ricci
et al. 2015 and Perera et al. 2014 and Feng et al. 2017)
In this paper our focus is on Cognitive IoT application
in smart home for automated recognition of
inhabitant activities to enhance the independent living
experience to improve daily quality of life. In smart
home scenario, obtained sensor data of inhabitants
activities interactions sequences within the
environment is segmented and can be labeled as
specific activity instance with description. The
detected activity instances are used to train an activity
recognition model as a result, the trained model are
Jamnal, G. and Liu, X.
A Cognitive-IoE Approach to Ambient-intelligent Smart Home.
DOI: 10.5220/0006304103020308
In Proceedings of the 2nd International Conference on Internet of Things, Big Data and Security (IoTBDS 2017), pages 302-308
ISBN: 978-989-758-245-5
Copyright © 2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
then used to classify and assign a label to new activity
instance. As Nazerfard et al. (2010) argued that
discovering the order of activities can be effectively
used for predicting the next activity in a home
automation system using their temporal relation
information. The activities pattern can be recognized
by using machine-learning techniques such as HMM,
Navies Bayes, Decision Tree, ANN and SVM and
KNN etc. (Fahad et al. 2014 and Somov et al. 2013
and Wu et al. 2014)
The main aim of CIoT in smart home environment is
to improve quality of life via developing a ambient
intelligent living environment. The communication
layer of multiple sensors can control home appliance
via actuator/device controller to help inhabitant into
the daily activities. In other words, CIoT works as
brain, where raw data gathered from sensors and
information collected and fused into decision making
unit, for computing controlling commands to achieve
specific goals. In such cases every home appliance
can be programmed according to inhabitant need and
living patterns. In industry, we have few smart
solutions such as smart grids, electric meters, security
controls system, lighting system, which can be
programmed to customized as per individual desire
(Feng et al.2017).
The research project aims to develop a novel dynamic
architecture including its related models and
mechanism to Cognitive-IoE based smart homes,
where the functionality of ambient intelligence is
extended towards more proactive possibilities, i.e.,
the smart environment not only monitors
people/devices for tasks, or support them by
executing their requests, but also influence and
change their plans and intentions. In the dynamic
environment, a home is equipped with multiple
sensor (motion, light, noise) to perceive the
environmental data in consistent/inconsistent state
and preprocessed for further activity (similar/non-
similar) classification. Human activity sequences can
be analyzed from sensor data using their temporal
values and transferred to an inference engine to
recognized their daily activity patterns as routine,
location and social contexts.
The rest of this paper is organized as follows. Section
2 summaries related work on activity recognition
done in the community. A bottom up approach to
inhabitant activity recognition in smart space is
presented as the ACM architecture Section 3. The
various classification based evaluation methods are
presented in Section 4. The work is concluded in
Section 5 and future work is discussed in Section 6.
In recent years, learning and understanding the
observed activity and event mining are the central
research area to smart home studies. Activity refers to
complex behaviors consisting of a sequence of action
and overlapped action that can be performed by a
single individual or several individuals interacting
with each other. Some significant smart home activity
recognition research work has been done in Care-lab,
CASAS, Grator-Tech HIS, Aware Home, iDorm, and
MavHom projects. In particular, the process of
activity recognition can be divided in four steps such
as i) sensing, ii) data-preprocessing, iii) data
modelling Feature extraction and iv) feature
selection. The major research work is in progress by
the tech giant IBM Watson, where cognitive
appliances talk to each other and the central
computing unit works as personalized digital assistant
for granting access and controlling various appliances
example e.g. smart locks, digital reminders, etc. IBM
Watsons cognitive IoT vision is to create a custom
tailored environment for individual residents by
adapting their preference and patterns, which not only
ensure better security, predictive maintenance tasks
and alert system but also saves time and money of
individuals, working as personal assistants (IBM
Watson, 2017).
2.1 Subtractive Clustering: Pattern
Recognition In Smart Home
While we talk about ambient intelligent smart home,
the need of clustering methods arises to find
relationships between observed datasets. As we
know, clustering based classification is a well-known
approach to extract knowledge from obtained datasets
by dividing datasets into discreet classified clusters.
Two major clustering methods, K-means and the
subtractive clustering can resolve the problem of
separating similar and non-similar activities from
given datasets (sensors data). The k-means clustering
algorithms works on pre-segmented (known) clusters
numbers, where we assume the number of clusters in
advance for given datasets. Sometimes, it is not the
case to have prior knowledge about the required
A Cognitive-IoE Approach to Ambient-intelligent Smart Home
number of clusters for obtained data sets. Also it
makes system less flexible to identify appropriate
cluster due to limited number of clusters. On the other
hand, to improve the learning ability we can take the
help of subtractive clustering algorithms to find
patterns in inhabitant behavior. As figure 1 shows,
subtractive clustering methods estimate the cluster
center and select the data point with highest potential
value to be the first cluster center, and later remove
all data points within the vicinity of the first cluster
center in order to determine the next data cluster and
its center location. The process keeps going on until
all the remaining datasets identify their radius of a
cluster center. Sub-clustering is a quick one pass
algorithm, moreover it works in the context that the
expert has no idea about the cluster number and
specifications. In the same way, the smart home
control system depends on the simulation of human
experience to make itself an intelligent control system
to perceptually make judgement to the external
environment variables, which are very strong,
moderate, and very weak (Amirjavid et al. 2014 and
Fahad et al. 2014).
Figure 1: Subtractive clustering for data pre-processing.
2.2 Hidden Markov Model: Pattern
Recognition In Smart Home
HMM is the most generative temporal probabilistic
model, It is applied to find hidden states (y1, y2, . . . ,
yT), in the observed time series data sequence (x1, x2,
. . . , xT). HMM is fundamentally based on the
independence of hidden states and the condition
independence of observation parameters stipulating
that p(xt|yt, x1, x2 . . . xt- 1, y1, y2 . . . yt1) = p(xt|yt).
The observable state at time t, xt, depends only on the
current hidden state yt. That is, the probability of
observing xt while being at yt is independent of all
other observable and hidden variables. The joint
probability p(x, y) of the observations and hidden
states can be factorized as follows:
The links between the hidden states are labeled with
the transition probabilities and those between the
hidden states and the observed sequence are labeled
with the emission probabilities. Using the initial and
transition probability matrix of the observable
Markov model, it is possible to calculate the
probability of any new activity sequences from the
current activity sequence. That is the model can
estimate the probabilities of new sequence, as shown
in figure 2, HMM can solve three problems such as:
Evaluation, Decoding and Learning problem. The
evaluation problem uses a forward-backward
algorithm to evaluate the probability of efficient
computation based on a given set of observations. The
learning problem optimizes parameters of a model to
better describe the observations using the
expectation-maximization (EM) algorithm. Finally,
the decoding problem uses the Viterbi algorithm to
find the most likely state sequence given an
observation sequences. (Benmansour et al. 2016)
Figure 2: Problem solving stages in HMM. (Benmansour et
al. 2016).
As HMM jobs to identify the most likelihood
sequences in given observed data vectors, so it is a bit
challenging to adjust or re-estimate the model
parameters to obtain the maximized probability of
likelihood sequence vector. Therefore, an iterative
approach of Baum-welch algorithm can be applied to
re-estimate the HMM parameters sub-optimally, as
expectation maximization(EM) method. Steps are :
(1) λ
= initial model
(2) compute new λ based on 𝜆
and given observation
sequence 𝑥
(3) if log Pr(𝑥/𝑦) log Pr(𝑥/𝜆
) < Γ stops, else set
= λ and repeat step (2)
Here, Γ represents the minimum tolerance value
between two subsequent model.
IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security
Afterwards, the forward-backward algorithm places
the re-estimated parameter in set of equations as
Here, Q=states sequences (q1,q2, ..., qk)
λ = HMM model
= input data vector<x1,x2,x3, xk> of
observation sequence.
(Hassan et al. 2013)
In particular, when we want to apply machine
learning algorithms for activity pattern recognition in
smart home scenario, we follow the sequential
occurrence of regular activity in order to find changes
in patterns of individual lifestyle. In addition, there
are many methods for activity recognition in the
related area, therefore it become important to
consider general classification and examination of
each approach and their implementation constraints
for specific problem solution (Zolfaghari and
Keyvanpour, 2016). In our research work, we applied
the bottom-up approach for human activity
recognition based on the data-driven probabilistic
model. In contrast, the key contribution of our model
is to identify users key intentions based on observed
sensors data. We used, subtractive clustering and
Hidden Markov Model combination to recognize
activities patterns in smart homes. As we applied the
two level classification model in our proposed ACM
framework in figure 3, inhabitant activity recognition
is done in four stages; sensing, pre-processing, data
modelling and context relevancy check such as social,
environmental and activity intentions.
In the first step, the sensor raw data are collected,
containing information about objects, which have
temporal information of each activity event.
Furthermore, in order to separates the relationship
between similar and non-similar activities, a
subtractive clustering methods is applied to find
relevant clusters in datasets. The sub-clustering
algorithm works on radius parameter to identify the
number of clusters. As a result, the large set of data
problem is subdivided into the small datasets
problem. Afterwards, using initial transition
Figure 3: Proposed ACM (Ambient-Cognition Model)
framework components.
probability and emission matrix of the observed
sequences, we applied Baum-welch algorithm to re-
estimate the HMM probability (transition and
emission) matrix input parameters. later the ACM
trained for 111 datasets using a re-estimated
probability matrix.
We applied Viterbi algorithm to compute most
likelihood sequence (probable-path) to find the
maximum occurrence over all possible state
sequences of given observed sequence datasets,
which make our ACM model more ambient
intelligent and proactive to understand inhabitant
intentions. Furthermore, likelihood sequence is
evaluated comparing to real time observed date
sequence to check system accuracy and intelligence
to identify recognized patterns. The following Viterbi
equation can find most likely sequences of hidden
states of given observations:
Furthermore, we identify the users intentions from
HMM likelihood sequences, we can subdivide them
into further three categories such as activity intention
of users movements for specific work, and location
intention of users specific movement and appearance
on specific locations. In particular, a routine intention
to identify users daily activities can be outlined with
repetitive task performed. Along with this, the
activity knowledge base will be storing intentions for
historical information purpose to link up for future
A Cognitive-IoE Approach to Ambient-intelligent Smart Home
In an ambient environment, ADL (Activity daily
living) can be evaluated in many different ways such
as training and testing of each routine activity.
Besides, there might be possibility of data scarcity of
each activity, which might create problem of
accuracy and performance evaluations. Above all,
major problem might occur when inhabitant is
reluctant to reveal their behavioral data due to privacy
and ethical consideration. Therefore, a researcher
should consider the availability of activities data sets
before analyzing human activity recognition in
ambient intelligent smart spaces.
For our research, we used existing ADL (activity
daily living) datasets of multiple ubiquitous sensors
placed in various location provided by MIT media
Lab from Tapia (2003) research work. We will be
using provided activity sequence of 111 samples of
single user activity which are labeled based on each
activity type.
Table 1: Activity daily living with their unique ID.
Activity Daily Living
Preparing dinner
Watching TV
Preparing breakfast
Preparing lunch
Taking medication
talking on phone
However, the main problem in MIT media lab data
set is that, no time interval has been set in
experimental data sets. As a result, number of activity
states are not equal for each day. For example, some
days have more activities sensors values compare to
other days (day1 captured 5 activities, day2 have 9
activities captured). It could have been better if value
0 is provided where no activities event happened for
specific day to maintain number of activity states in
correct sequence manner. Hence, all days would have
equal set of data value in matrix. In other words,
every row has the same length. The below figure 4,
represents the 10 activities (mentioned in table 1) of
111 observed sequences from MIT media lab during
19/4/2003 to 31/4/2003.
Figure 4: MIT media lab activity datasets.
Furthermore, these 10 household activities are sub-
clustered into 3 main activities clusters such as
personal needs (1), domestic work (2) and relaxing
(3) labels, mentioned in below table 2.
Table 2: three-activity cluster of household.
Activity Cluster
Personal needs(1)
Taking medication
talking on phone
Domestic work(2)
Preparing lunch
Preparing dinner
preparing breakfast
Watching TV
Matlab 2016 version were used for test experiments,
furthermore 111 observed activity datasets have been
labeled into 3 main activities. Initial transition and
emission matrix has been defined based on prior
knowledge base. Afterwards, using Baum-welch
algorithm further re-estimation of transition and
emission probability matrix has been successfully
achieved. Correspondingly, Matlab train the system
for 111 observed sequences and Viterbi algorithm
identified most likelihood activity sequences with the
IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security
77% of accuracy, shown in figure 5 as Matlab
simulation result.
Figure 5: HMM likelihood activity sequence compares to
observed sequence.
This paper proposed a new type of ambient intelligent
architecture for smart homes to understand the
inhabitant intentions proactively. combining
knowledge from traditional context aware pervasive
systems with modern era Cognitive IoT technologies.
The building block of ACM architecture is well
equipped with artificial intelligence machine learning
algorithms of subtractive clustering and Hidden
Markov model to recognize patterns in daily activity
living. ACM ensure the comfortable living
environment for inhabitant as its primary goal.
Moreover, the ACM model can be successfully
applied in elderly health care systems, smart
classrooms, and smart spaces for automated
environment. While given MIT media lab data sets
itself a detailed activities observation of inhabitants
but could have been lot more better if have more data
samples are available to avoid the data scarcity
problem. In addition, the provided MIT media lab
datasets could have better attributes in well-organized
time set intervals. However, our ACM model well
trained over provided data samples and successfully
achieved 77% of likelihood sequence accuracy.
As part of our ongoing work, we are extending our
ACM to adapt a fuzzy-rule based system for
executing tasks for physical world. We plan to
combine ACM with device controller by inferring
fuzzy rule activations. Thus, it will be an excellent
device controller for all appliances in smart
In addition, we would be creating our own smart
home lab, where sensors data will be collected in
predefined time set interval to maintain data accuracy
and uniformity. As a result, the data scarcity problem
would be avoided automatically and the result would
be simulated with increased pattern recognition
Amirjavid, F., Bouzouane, A., & Bouchard, B. (2014). Data
driven modeling of the simultaneous activities in
ambient environments. Journal of Ambient
Intelligence and Humanized Computing, 5(5), 717-
Benmansour, A., Bouchachia, A., & Feham, M. (2016).
Multioccupant activity recognition in pervasive smart
home environments. ACM Computing Surveys
(CSUR), 48(3), 34.
Fahad, L. G., Tahir, S. F., & Rajarajan, M. (2014). Activity
recognition in smart homes using clustering based
Feng, S., Setoodeh, P., & Haykin, S. (2017). Smart Home:
Cognitive Interactive People-Centric Internet of
Things. IEEE Communications Magazine, 55(2), 34-
Hassan, M. R., Ramamohanarao, K., Kamruzzaman, J.,
Rahman, M., & Hossain, M. M. (2013). A HMM-based
adaptive fuzzy inference system for stock market
forecasting. Neurocomputing, 104, 10-25.
IBM Watson, (2017) Watson Internet of Things [Online]
buildings/cognitive-systems-in-buildings/ [accessed
MIT (2003) Using Machine Learning for Real-time
Activity Recognition and Estimation of Energy
Expenditure [Online] <http://alumni.media.mit.edu
[Accessed 12/01/2017].
Munguia Tapia, E. (2003). Activity recognition in the
home setting using simple and ubiquitous
sensors (Doctoral dissertation, Massachusetts Institute
of Technology).
Nazerfard, E., Rashidi, P., & Cook, D. J. (2010, December).
Discovering temporal features and relations of activity
patterns. In 2010 IEEE International Conference on
Data Mining Workshops (pp. 1069-1075). IEEE.
Perera, C., Liu, C. H., Jayawardena, S., & Chen, M. (2014).
A survey on internet of things from industrial market
perspective. IEEE Access, 2, 1660-1679.
Ricci, A., Piunti, M., Tummolini, L., & Castelfranchi, C.
(2015). The Mirror World: Preparing for Mixed-Reality
Living. IEEE Pervasive Computing, (2), 60-63.
A Cognitive-IoE Approach to Ambient-intelligent Smart Home
Satyanarayanan, M. (2001). Pervasive computing: Vision
and challenges.Personal Communications, IEEE, 8(4),
Somov, A., Dupont, C., & Giaffreda, R. (2013). Supporting
Smart-city Mobility with Cognitive Internet of Things,
Wu, Q., Ding, G., Xu, Y., Feng, S., Du, Z., Wang, J., &
Long, K. (2014). Cognitive Internet of Things: A New
Paradigm beyond Connection. IEEE Internet of Things
Journal, PP(99), 11. http://doi.org/10.1109/JIOT
Zolfaghari, S., & Keyvanpour, M. R. (2016). SARF: Smart
activity recognition framework in Ambient Assisted
Living. In Computer Science and Information Systems
(FedCSIS), 2016 Federated Conference on (pp. 1435-
1443). IEEE.
IoTBDS 2017 - 2nd International Conference on Internet of Things, Big Data and Security