SMART AND INTERACTIVE FUTURE HOMES
Integration of Autonomic Computing and New HCI Methods
Rafael Del-Hoyo, Luis Miguel Sanagustín, Carolina Benito, Isabelle Hupont and David Abadía
Instituto Tecnológico de Aragón, P.T. Walqa Ctra. Zaragoza, N-330a, Km 566, Cuarte (Huesca), Spain
Keywords: Autonomic Computing, HCI, Virtual Agents, Fuzzy Logic, Multimedia.
Abstract: Nowadays, huge R&D efforts are running on the re-invention of the Internet so that it is able to cope with
future challenges, like the viral growth of the number of connected users, devices, services and user-
generated contents. Today’s houses are slowly turning into a complex electronic net of devices. The
increasing complexity of systems and the need for these systems to remain simple, accessible and
transparent for the user, makes it necessary to research technologies that enable intelligent and autonomous
computing and new ways of interacting with future home. Autonomic computing systems are those which
can manage themselves given high level objectives. If we integrate autonomic computing and new
interactive user mechanisms like virtual agents, we obtain the future smart homes. These houses would
detect the people inside, and self-configure by personalizing the services for each user and detecting new
devices plugged to the house: would self-optimize by disconnecting lights or closing doors if people aren’t
present: would self-heal by controlling sensors to prevent problems related to physical and software
elements; and would self-protect by identifying the current users at home, and preventing external attacks.
1 INTRODUCTION
Nowadays, huge R&D efforts are running on the re-
invention of the Internet so that it is able to cope
with future challenges, like the viral growth of the
number of connected users, devices, services and
user-generated contents. Today’s houses are slowly
turning into a complex electronic net of devices.
Multimedia TVs based in DLNA (Digital Living
Network Alliance), sensors, automation controls and
energy consumption meters are connected to the
Internet via residential gateways. In a close future
people is going to be immersed in the Internet of
things or the internet of objects connected to the
cloud. The increasing complexity of systems and the
need for these systems to remain simple, accessible
and transparent for the user, makes it necessary to
research technologies that enable intelligent and
autonomous computing and new ways of interacting
with future home.
Autonomic computing systems are those which
can manage themselves given high level objectives.
These systems include environments that are able to
evolve without the need for human interaction.
These environments are capable of installing,
configuring, maintaining and healing themselves,
and their own components.
This paper presents an Autonomic Interactive
Fusion engine platform developed in the GENIO
project (http://projects.celtic-initiative.org/genio/).
The main element of the architecture is the
Intelligent Autonomous System in charge of the
information fusion process. This also controls a
virtual agent that allows increases the interactivity
from user perspective.
This paper is structured in the following way; the
first section is a brief statement about autonomic
computing and virtual agent in smart homes. The
second section describes current projects in smart
homes and GENIO project and the following section
describes the intelligent framework used for smart
homes. The fourth section explains how the
information is joined inside the fusion information
engine. Finally, several conclusions are set forth.
1.1 Autonomic Computing
The essence of autonomic computing systems is
self-management, or, the ability to reduce human
interaction in administration tasks to the minimum.
As it’s explained in previous researchs, these
507
Del-Hoyo R., Miguel Sanagustín L., Benito C., Hupont I. and Abadía D..
SMART AND INTERACTIVE FUTURE HOMES - Integration of Autonomic Computing and New HCI Methods.
DOI: 10.5220/0003748805070510
In Proceedings of the 4th International Conference on Agents and Artificial Intelligence (ICAART-2012), pages 507-510
ISBN: 978-989-8425-95-9
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
systems should provide Self-configuration, self-
optimization, self-healing, self-protection. The
system should incorporate itself seamlessly, and the
other components present in the system must adapt
to its presence by learning new configurations or
topologies. An automatic system should continually
seek ways to tweak parameters, and, at the same
time, should be able to find and apply the lastest
updates for each system component. Autonomic
systems should detect, trace, diagnose and repair
bugs and failures. Autonomic systems should defend
themselves from large scale problems arising from
malicious attacks or big failures.
1.2 Human Interaction through Virtual
Agents
Virtual agents have proved to be a useful way of
HCI. For humans, it is easier to communicate with a
computer through a conversation with a virtual agent
as opposed to just a keyboard and mouse. The
virtual agent can be a realistic 3D representation of a
human being, but can also be a 3D cartoon or just a
2D animated agent. This depends on several factors
such as the kind of application or the target user.
Virtual agents have been used in very different
contexts, such as marketing, education, shopper
assistants, or personal trainers.
Firstly, natural human-human interaction is
multimodal: we communicate through speech and
use body language (posture, facial expressions,
gaze) to express affect, mood, attitude and attention.
Thus, when communicating with each other, human
beings have to process and react in real-time to a
broad spectrum of data coming from different
channels: visual, auditory, tactil senses. To make a
virtual agent interact in a consistent, emotionally
empathic and intelligent way with the user, a
strategy must be defined for recognizing, integrating
and interpreting user information coming from
different modalities (video, audio, etc.).
Secondly, it is important to realize how the
human mind works to correctly “model” the virtual
agent’s reasoning mechanisms. The human brain is
characterized by its capacity to handle and store
uncertain and confusing perceptions. People usually
face problems with great uncertainty and partial,
context-dependent, and contradictory information.
Softcomputing techniques, in special Fuzzy Logic,
make it possible to model these types of problems
and to find solutions similar to the ones taken by
human beings. In doing so, it is possible to develop a
more “cognitive” computation that tackles
effectively the interaction among persons and virtual
agents, how they communicate and act through
words and perceptions.
Finally, the virtual agent must be believable: it
has to move properly, paying special attention to its
facial expressions and body gestures, and have the
capacity to talk in natural language (Cowie, 2000).
Emotions have been proved to play an essential role
in decision making, perception, learning and more
(Egges, 2004). Consequently, besides its external
appearance, the virtual agent must possess some
affectivity, an innate characteristic in humans, for
which it is necessary to carefully manage the
emotional display of the virtual agent.
Human Computer Interaction (HCI) gets more
natural when using a virtual agent as computer side
communication entity. Thanks to both, verbal and
non-verbal communication, the interaction between
the user and the virtual agent becomes more
credible.
1.3 SMART Interactive MEDIA
HOMES
One of the most important fields to apply
Autonomic Computing and Human Computer
Interaction technologies is houses, thus making them
intelligent or smart houses. These houses would
detect the people inside, self-configure by
personalizing the services for each users and
detecting and configuring new devices plugged into
the house; would self-optimize by disconnecting
lights or closing doors if people aren’t present;
would self-heal by controlling sensors and
preventing problems related to physical and software
elements; and would self-protect by identifying the
current users at home, and preventing
external
attacks.
The architecture is rapidly retargeted to a
specific configuration. The engine can also self-heal,
when a device or service is removed or fails, the
system should adapt itself in order to offer its
services in an alternative way to reduce the impact
of the device loss. At last, the system can self-adapt,
because users’ needs are different for each user at
any given moment, the system should adjust its
services in order to fulfil user preferences. The
University of Colorado has introduced the adaptive
house. They present the idea of adapt and
reconfigure their autonomic system by observing the
lifestyle and desires of the inhabitants, and learning
to anticipate and accommodate their needs. The
autonomic system monitors the environment,
observing the actions taken by its occupants, and it
uses neural network reinforcement learning and
ICAART 2012 - International Conference on Agents and Artificial Intelligence
508
prediction techniques to infer patterns in the
environment that predict these actions and perform
them automatically. This system also introduces
self-optimization by trying to conserve energy
sources, where possible. For example, it can predict
when the occupants will return home, and determine
when to start heating the house, detect patterns of
hot water usage to disconnect the water heater at
times that is never used, control lighting patterns and
intensities based on occupant activities, etc..
2 FRAMEWORK TECHNOLOGY
OVERVIEW
The aim of this section is to establish the Intelligent
Autonomous System’s technology overview. One of
the possible technologies which is candidate for
handling this type of smart home is algorithm based
on soft-computing/computational intelligence
techniques (Jang, 1997). These algorithms are able
to work with a great number of data (even noisy and
incomplete), and they also allow predicting the
behaviour of highly nonlinear systems, as is the case
of Home Systems and in special communication
systems. As we have in GENIO project, these
properties allow us to analyse, predict the state of an
IP network or to make decision about any problem
inside of the home. In this project, the development
of an intelligent decision support system (iDSS) is
proposed, called Intelligent Autonomous System
(IAS), based on some well-known artificial neural
models (Haykin, 1999). IAS will permit us to
manage entire home status, dealing with high
dimensionality information, through a new advanced
interface, a virtual Agent. The objective of this
virtual agent is to be the human-interface between
the home user and the autonomous systems, inform
the user about the status of the home, and the
predicted situations found. Also, thanks to this
interface the user can manage the home devices
using several other interfaces, like SNMP or uPNP.
The control logic in the IAS platform is
implemented inside of the intelligent framework
using natural language rules (fuzzy rules) and neural
network for pattern recognition. This framework is
also responsible of the behaviour of the virtual
agent. This IAS platform is able to evolve and adapt
according to the actions obtained from the user.
During the learning process user patterns like the
number of user repetitions, watching movies or
managing home devices are learned by means of a
neural network supervised learning process.
Thanks to the rules-based intelligent framework,
the proposed platform is a powerful tool for general
autonomous home systems. The platform can be
adapted easily to any kind of device, network
condition or content, and the exercises can be
designed by any person even if he or she does not
have programming knowledge, thanks to the natural
language rules programming. The main AI
technologies uses in the decision autonomous
system for GENIO are neural networks, Rule
Engines and finally AIML
2.1 The Intelligent Support Interaction
System (ISIS)
ISIS is the main element of the proposed application.
It is the evolution of a softcomputing-based
intelligent system called PROPHET that enables
real-time automatic fuzzy decision making and self-
learning over any kind of incoming inputs (from
sensors, video channels, audio channels, probes…).
The system has already been successfully used in
different domains such as logistics decision making
systems (Martínez 2011). ISIS is the engine in
charge of the logic of the platform from IAS point of
view. It is also the inference engine that makes the
virtual Agent to react to different inputs coming to
the platform. According to different inputs of the
platform, ISIS extracts knowledge and thanks to the
use of Neuro-Fuzzy techniques (Lin and Lee, 1996).
ISIS consists of a set of modules for pre-
processing, integrating and extracting information
and making decisions in a flexible way under
uncertain contexts. The system is based on a state
machine: each module generates events that are
treated asynchronously inside the state machine.
The high level Autonomous System design can be
described in the following modules:
Hybrid rule inference engine: it is the main sub-
system of the Autonomous System. It is in charge of
rule-based decision-making tasks. It is a hybrid rule
inference engine since it can both deal with crisp
rules (applied to exact inputs’ values) and execute
inference from rules that handle fuzzy concepts.
Knowledge and data persistence module: system
that manages data and knowledge (rules)
information.
Integration and transformation module: module
in charge of filtering, synchronizing and pre-
processing the incoming inputs, in order to make
them compatible with the hybrid rule inference
engine input format.
Application control module: state machine that
controls the Autonomous Intelligent System
SMART AND INTERACTIVE FUTURE HOMES - Integration of Autonomic Computing and New HCI Methods
509
behavior.
AIML module: the AIML module computes an
appropriate natural language answer, for a given
user interaction context.
Communication interface with the GUI: interface
that manages communication between the
Autonomous System and the GUI.
Communication interface with the Topology
Manager: component that both informs the
Autonomous System in real-time about the events
that occur in the home network and allows to send
configuration commands to connected home devices
on the basis of the decisions taken by the
Autonomous System
3 MULTIMODAL FUSION
ENGINE
Currently, the system integrates data from the
following sources:
Pattern Recognition: through the inclusion of
different classification algorithms previously
predefined. This preprocess module allows the
generation of new attributes based on this data
mining algorithm such as classification of non-
linear patterns and the generation of an attribute
that represents the output of this classifier.
Virtual Agent Speech Recognition: based on
the recognition of the user's voice, it is
converted into text and placed as an attribute of
the text itself. In the inference engine are
inserted the text converted, the value of
accuracy of this text and the AIML engine
response from this text.
Presence recognition and number of people
counting: the number of persons found in the
scene.
Gesture Recognition: gesture recognition is
performed by a gesture recognition algorithm
within the platform Kinect.
Feature Image Classification Algorithm: With
a vision algorithm based on a feature detection
algorithm called Surft.
Information about UPnP devices / network
existing inside the home: when an uPnP device
is detected, ISIS dynamically generates a
number of attributes into the system
corresponding to each one properties associated
to UPnP device.
SNMP information from network devices
Home: ISIS is able to monitor any home
network device.
Energy consumption Information and home
automation sensors: home sensors are mapped
to numeric attributes inside the inference engine.
These attributes are generated at each time t and
they can be complemented with new attributes
generated by crisp rules. This information is mapped
into the working memory and based on this
information are fired the rules in the RETE inference
engine.
4 CONCLUSIONS
A novel platform for creating smart interactive
homes has been presented. The platform can be seen
as an prototype in GENIO project. From
Autonomous Computing point of view, the
possibility of creating a Self-configuration system
capable of detect uPNP devices dynamically. Using
fuzzy rules inside infrence engine is possible a Self-
optimization and Self-healing of the Home. Finally
the integration and information fusion and the
pattern recoginiton feature allows the Self-protection
and self-healing fore diagnosis.
ACKNOWLEDGEMENTS
This work has been partly financed by the GENIO
project (TSI-020400-2010-98), funded by the
Spanish Ministry of Industry, project FEDER ATIC,
and the Grupo de Ingeniería Avanzada (GIA) of the
Instituto Tecnológico de Aragón.
REFERENCES
Abraham, A., (2001), Neuro fuzzy systems: state-of-the-art
modeling techniques, in: Proceedings of IWANN2001,
Lecture Notes on Computer Science, vol. 2084,
Springer, Berlin, pp. 269–276.
Cowie, R. Douglas-Cowie, E., and Schroeder, M., (2000)
Speech and Emotion. A conceptual framework for
research, Proc. of the ISCA Workshop, Belfast.
Egges, A. Kshirsagar, S. and Magnenat-Thalmann, N.,
(2004), Generic personality and emotion simulation
for conversational agents, Computer Animation and
Virtual Worlds, vol. 15, 2004, pp. 1–13.
Haykin, S., (1999) Neural Networks: A Comprehensive
Foundation, second ed, Prentice-Hall, Englewood
Cliffs, NJ.
Martínez, H, Del-Hoyo, R, Sanagustín, L, Hupont I,
Abadía. D., Sagüés C.: (2011) Augmented Reality
based Intelligent Interactive e-Learning Platform.
ICAART (1) 2011: 343-348
ICAART 2012 - International Conference on Agents and Artificial Intelligence
510