SMART AND INTERACTIVE FUTURE HOMES

Integration of Autonomic Computing and New HCI Methods

Rafael Del-Hoyo, Luis Miguel Sanagustín, Carolina Benito, Isabelle Hupont and David Abadía

Instituto Tecnológico de Aragón, P.T. Walqa Ctra. Zaragoza, N-330a, Km 566, Cuarte (Huesca), Spain

Keywords: Autonomic Computing, HCI, Virtual Agents, Fuzzy Logic, Multimedia.

Abstract: Nowadays, huge R&D efforts are running on the re-invention of the Internet so that it is able to cope with

future challenges, like the viral growth of the number of connected users, devices, services and user-

generated contents. Today’s houses are slowly turning into a complex electronic net of devices. The

increasing complexity of systems and the need for these systems to remain simple, accessible and

transparent for the user, makes it necessary to research technologies that enable intelligent and autonomous

computing and new ways of interacting with future home. Autonomic computing systems are those which

can manage themselves given high level objectives. If we integrate autonomic computing and new

interactive user mechanisms like virtual agents, we obtain the future smart homes. These houses would

detect the people inside, and self-configure by personalizing the services for each user and detecting new

devices plugged to the house: would self-optimize by disconnecting lights or closing doors if people aren’t

present: would self-heal by controlling sensors to prevent problems related to physical and software

elements; and would self-protect by identifying the current users at home, and preventing external attacks.

1 INTRODUCTION

Nowadays, huge R&D efforts are running on the re-

invention of the Internet so that it is able to cope

with future challenges, like the viral growth of the

number of connected users, devices, services and

user-generated contents. Today’s houses are slowly

turning into a complex electronic net of devices.

Multimedia TVs based in DLNA (Digital Living

Network Alliance), sensors, automation controls and

energy consumption meters are connected to the

Internet via residential gateways. In a close future

people is going to be immersed in the Internet of

things or the internet of objects connected to the

cloud. The increasing complexity of systems and the

need for these systems to remain simple, accessible

and transparent for the user, makes it necessary to

research technologies that enable intelligent and

autonomous computing and new ways of interacting

with future home.

Autonomic computing systems are those which

can manage themselves given high level objectives.

These systems include environments that are able to

evolve without the need for human interaction.

These environments are capable of installing,

configuring, maintaining and healing themselves,

and their own components.

This paper presents an Autonomic Interactive

Fusion engine platform developed in the GENIO

project (http://projects.celtic-initiative.org/genio/).

The main element of the architecture is the

Intelligent Autonomous System in charge of the

information fusion process. This also controls a

virtual agent that allows increases the interactivity

from user perspective.

This paper is structured in the following way; the

first section is a brief statement about autonomic

computing and virtual agent in smart homes. The

second section describes current projects in smart

homes and GENIO project and the following section

describes the intelligent framework used for smart

homes. The fourth section explains how the

information is joined inside the fusion information

engine. Finally, several conclusions are set forth.

1.1 Autonomic Computing

The essence of autonomic computing systems is

self-management, or, the ability to reduce human

interaction in administration tasks to the minimum.

As it’s explained in previous researchs, these

507

Del-Hoyo R., Miguel Sanagustín L., Benito C., Hupont I. and Abadía D..

SMART AND INTERACTIVE FUTURE HOMES - Integration of Autonomic Computing and New HCI Methods.

DOI: 10.5220/0003748805070510

In Proceedings of the 4th International Conference on Agents and Artiﬁcial Intelligence (ICAART-2012), pages 507-510

ISBN: 978-989-8425-95-9

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

systems should provide Self-configuration, self-

optimization, self-healing, self-protection. The

system should incorporate itself seamlessly, and the

other components present in the system must adapt

to its presence by learning new configurations or

topologies. An automatic system should continually

seek ways to tweak parameters, and, at the same

time, should be able to find and apply the lastest

updates for each system component. Autonomic

systems should detect, trace, diagnose and repair

bugs and failures. Autonomic systems should defend

themselves from large scale problems arising from

malicious attacks or big failures.

1.2 Human Interaction through Virtual

Agents

Virtual agents have proved to be a useful way of

HCI. For humans, it is easier to communicate with a

computer through a conversation with a virtual agent

as opposed to just a keyboard and mouse. The

virtual agent can be a realistic 3D representation of a

human being, but can also be a 3D cartoon or just a

2D animated agent. This depends on several factors

such as the kind of application or the target user.

Virtual agents have been used in very different

contexts, such as marketing, education, shopper

assistants, or personal trainers.

Firstly, natural human-human interaction is

multimodal: we communicate through speech and

use body language (posture, facial expressions,

gaze) to express affect, mood, attitude and attention.

Thus, when communicating with each other, human

beings have to process and react in real-time to a

broad spectrum of data coming from different

channels: visual, auditory, tactil senses. To make a

virtual agent interact in a consistent, emotionally

empathic and intelligent way with the user, a

strategy must be defined for recognizing, integrating

and interpreting user information coming from

different modalities (video, audio, etc.).

Secondly, it is important to realize how the

human mind works to correctly “model” the virtual

agent’s reasoning mechanisms. The human brain is

characterized by its capacity to handle and store

uncertain and confusing perceptions. People usually

face problems with great uncertainty and partial,

context-dependent, and contradictory information.

Softcomputing techniques, in special Fuzzy Logic,

make it possible to model these types of problems

and to find solutions similar to the ones taken by

human beings. In doing so, it is possible to develop a

more “cognitive” computation that tackles

effectively the interaction among persons and virtual

agents, how they communicate and act through

words and perceptions.

Finally, the virtual agent must be believable: it

has to move properly, paying special attention to its

facial expressions and body gestures, and have the

capacity to talk in natural language (Cowie, 2000).

Emotions have been proved to play an essential role

in decision making, perception, learning and more

(Egges, 2004). Consequently, besides its external

appearance, the virtual agent must possess some

affectivity, an innate characteristic in humans, for

which it is necessary to carefully manage the

emotional display of the virtual agent.

Human Computer Interaction (HCI) gets more

natural when using a virtual agent as computer side

communication entity. Thanks to both, verbal and

non-verbal communication, the interaction between

the user and the virtual agent becomes more

credible.

1.3 SMART Interactive MEDIA

HOMES

One of the most important fields to apply

Autonomic Computing and Human Computer

Interaction technologies is houses, thus making them

intelligent or smart houses. These houses would

detect the people inside, self-configure by

personalizing the services for each users and

detecting and configuring new devices plugged into

the house; would self-optimize by disconnecting

lights or closing doors if people aren’t present;

would self-heal by controlling sensors and

preventing problems related to physical and software

elements; and would self-protect by identifying the

current users at home, and preventing

external

attacks.

The architecture is rapidly retargeted to a

specific configuration. The engine can also self-heal,

when a device or service is removed or fails, the

system should adapt itself in order to offer its

services in an alternative way to reduce the impact

of the device loss. At last, the system can self-adapt,

because users’ needs are different for each user at

any given moment, the system should adjust its

services in order to fulfil user preferences. The

University of Colorado has introduced the adaptive

house. They present the idea of adapt and

reconfigure their autonomic system by observing the

lifestyle and desires of the inhabitants, and learning

to anticipate and accommodate their needs. The

autonomic system monitors the environment,

observing the actions taken by its occupants, and it

uses neural network reinforcement learning and

ICAART 2012 - International Conference on Agents and Artificial Intelligence

508

prediction techniques to infer patterns in the

environment that predict these actions and perform

them automatically. This system also introduces

self-optimization by trying to conserve energy

sources, where possible. For example, it can predict

when the occupants will return home, and determine

when to start heating the house, detect patterns of

hot water usage to disconnect the water heater at

times that is never used, control lighting patterns and

intensities based on occupant activities, etc..

2 FRAMEWORK TECHNOLOGY

OVERVIEW

The aim of this section is to establish the Intelligent

Autonomous System’s technology overview. One of

the possible technologies which is candidate for

handling this type of smart home is algorithm based

on soft-computing/computational intelligence

techniques (Jang, 1997). These algorithms are able

to work with a great number of data (even noisy and

incomplete), and they also allow predicting the

behaviour of highly nonlinear systems, as is the case

of Home Systems and in special communication

systems. As we have in GENIO project, these

properties allow us to analyse, predict the state of an

IP network or to make decision about any problem

inside of the home. In this project, the development

of an intelligent decision support system (iDSS) is

proposed, called Intelligent Autonomous System

(IAS), based on some well-known artificial neural

models (Haykin, 1999). IAS will permit us to

manage entire home status, dealing with high

dimensionality information, through a new advanced

interface, a virtual Agent. The objective of this

virtual agent is to be the human-interface between

the home user and the autonomous systems, inform

the user about the status of the home, and the

predicted situations found. Also, thanks to this

interface the user can manage the home devices

using several other interfaces, like SNMP or uPNP.

The control logic in the IAS platform is

implemented inside of the intelligent framework

using natural language rules (fuzzy rules) and neural

network for pattern recognition. This framework is

also responsible of the behaviour of the virtual

agent. This IAS platform is able to evolve and adapt

according to the actions obtained from the user.

During the learning process user patterns like the

number of user repetitions, watching movies or

managing home devices are learned by means of a

neural network supervised learning process.

Thanks to the rules-based intelligent framework,

the proposed platform is a powerful tool for general

autonomous home systems. The platform can be

adapted easily to any kind of device, network

condition or content, and the exercises can be

have programming knowledge, thanks to the natural

language rules programming. The main AI

technologies uses in the decision autonomous

system for GENIO are neural networks, Rule

Engines and finally AIML

2.1 The Intelligent Support Interaction

System (ISIS)

ISIS is the main element of the proposed application.

It is the evolution of a softcomputing-based

intelligent system called PROPHET that enables

real-time automatic fuzzy decision making and self-

learning over any kind of incoming inputs (from

sensors, video channels, audio channels, probes…).

The system has already been successfully used in

different domains such as logistics decision making

systems (Martínez 2011). ISIS is the engine in

charge of the logic of the platform from IAS point of

view. It is also the inference engine that makes the

virtual Agent to react to different inputs coming to

the platform. According to different inputs of the

platform, ISIS extracts knowledge and thanks to the

use of Neuro-Fuzzy techniques (Lin and Lee, 1996).

ISIS consists of a set of modules for pre-

processing, integrating and extracting information

and making decisions in a flexible way under

uncertain contexts. The system is based on a state

machine: each module generates events that are

treated asynchronously inside the state machine.

The high level Autonomous System design can be

described in the following modules:

Hybrid rule inference engine: it is the main sub-

system of the Autonomous System. It is in charge of

rule-based decision-making tasks. It is a hybrid rule

inference engine since it can both deal with crisp

rules (applied to exact inputs’ values) and execute

inference from rules that handle fuzzy concepts.

Knowledge and data persistence module: system

that manages data and knowledge (rules)

information.

Integration and transformation module: module

in charge of filtering, synchronizing and pre-

processing the incoming inputs, in order to make

them compatible with the hybrid rule inference

engine input format.

Application control module: state machine that

controls the Autonomous Intelligent System

SMART AND INTERACTIVE FUTURE HOMES - Integration of Autonomic Computing and New HCI Methods

509

behavior.

AIML module: the AIML module computes an

appropriate natural language answer, for a given

user interaction context.

Communication interface with the GUI: interface

that manages communication between the

Autonomous System and the GUI.

Communication interface with the Topology

Manager: component that both informs the

Autonomous System in real-time about the events

that occur in the home network and allows to send

configuration commands to connected home devices

on the basis of the decisions taken by the

Autonomous System

3 MULTIMODAL FUSION

ENGINE

Currently, the system integrates data from the

following sources:

• Pattern Recognition: through the inclusion of

different classification algorithms previously

predefined. This preprocess module allows the

generation of new attributes based on this data

mining algorithm such as classification of non-

linear patterns and the generation of an attribute

that represents the output of this classifier.

• Virtual Agent Speech Recognition: based on

the recognition of the user's voice, it is

converted into text and placed as an attribute of

the text itself. In the inference engine are

inserted the text converted, the value of

accuracy of this text and the AIML engine

response from this text.

• Presence recognition and number of people

counting: the number of persons found in the

scene.

• Gesture Recognition: gesture recognition is

performed by a gesture recognition algorithm

within the platform Kinect.

• Feature Image Classification Algorithm: With

a vision algorithm based on a feature detection

algorithm called Surft.

• Information about UPnP devices / network

existing inside the home: when an uPnP device

is detected, ISIS dynamically generates a

number of attributes into the system

corresponding to each one properties associated

to UPnP device.

• SNMP information from network devices

Home: ISIS is able to monitor any home

network device.

• Energy consumption Information and home

automation sensors: home sensors are mapped

to numeric attributes inside the inference engine.

These attributes are generated at each time t and

they can be complemented with new attributes

generated by crisp rules. This information is mapped

into the working memory and based on this

information are fired the rules in the RETE inference

engine.

4 CONCLUSIONS

A novel platform for creating smart interactive

homes has been presented. The platform can be seen

as an prototype in GENIO project. From

Autonomous Computing point of view, the

possibility of creating a Self-configuration system

capable of detect uPNP devices dynamically. Using

fuzzy rules inside infrence engine is possible a Self-

optimization and Self-healing of the Home. Finally

the integration and information fusion and the

pattern recoginiton feature allows the Self-protection

and self-healing fore diagnosis.

ACKNOWLEDGEMENTS

This work has been partly financed by the GENIO

project (TSI-020400-2010-98), funded by the

Spanish Ministry of Industry, project FEDER ATIC,

and the Grupo de Ingeniería Avanzada (GIA) of the

Instituto Tecnológico de Aragón.

REFERENCES

Abraham, A., (2001), Neuro fuzzy systems: state-of-the-art

modeling techniques, in: Proceedings of IWANN2001,

Lecture Notes on Computer Science, vol. 2084,

Springer, Berlin, pp. 269–276.

Cowie, R. Douglas-Cowie, E., and Schroeder, M., (2000)

Speech and Emotion. A conceptual framework for

research, Proc. of the ISCA Workshop, Belfast.

Egges, A. Kshirsagar, S. and Magnenat-Thalmann, N.,

(2004), Generic personality and emotion simulation

for conversational agents, Computer Animation and

Virtual Worlds, vol. 15, 2004, pp. 1–13.

Haykin, S., (1999) Neural Networks: A Comprehensive

Foundation, second ed, Prentice-Hall, Englewood

Cliffs, NJ.

Martínez, H, Del-Hoyo, R, Sanagustín, L, Hupont I,

Abadía. D., Sagüés C.: (2011) Augmented Reality

based Intelligent Interactive e-Learning Platform.

ICAART (1) 2011: 343-348

ICAART 2012 - International Conference on Agents and Artificial Intelligence

510