INTELLIGENT MULTI-DEVICE USER INTERFACES

Mario Vega Barbas and Juan R. Velasco

Department of Autom´atica, University of Alcala, Ediﬁcio Polit´ecnico, CTRA N-II. Km 31,600

28871 Alcal´a de Henares, Madrid, Spain

Keywords:

Interfaces, Human-computer interaction, Internet of things, Smart spaces, Proactive computing.

Abstract:

Interfaces represent the point of interaction between users and systems, deﬁning their adaptability level in

terms of usability and accessibility. These interfaces must advance in the same direction as the information

society, where all objects of the world around us are being introduced like services to create the Internet of

Things, and so, the Internet of The Future. Therefore, the new interfaces must guarantee that users have access

to all information and functionality that the new Internet can provide. Those advances require a new develop

interface concept where systems can create interfaces according to applications and users needs, always in a

proactive form. In this way, the work presented in this paper tries to deﬁne a new semantic and architectural

model to create intelligent and multi-device interface which may be adapted proactively according to the

context and each users requirements.

1 INTRODUCTION

The continuous evolution of multimedia technology

and Internet has enabled the Web to evolve to a new

way of communication where users, not necessary ex-

pert, can create and use vast amounts of information

of different nature (Web 2.0). The next evolutionary

leap in the information society and communications

will consist of the inclusion of objects in the world

around us as direct members of this society, so that

users can not only generate or consume information,

but may shape the environment in which they live to

improve their quality of life (DiY-SE, 2008). These

new objects and servers network, Internet of the Fu-

ture (CERP, 2009), aims to create dynamic informa-

tion networks made up of millions of interconnected

elements so they can process ubiquitous and intelli-

gent information around them, helping to create smart

environments.

In this way, due mainly to the wide range of user

proﬁles that can interact with this new Internet, the

design and use of interfaces for human-system in-

teraction becomes vital. There are now many tools

and technologies that enable users to make and com-

bine devices and serves such as XUI (Organic User

Interface) and NUI (Natural User Interface) (Bux-

ton, 2007) (Saffer, 2008), but these techniques require

deep technical expertise very speciﬁc. There are also

tools to automatically create interfaces for speciﬁc ap-

plications (Krzystof and Daniel, 2004) (Krzystof and

Daniel, 2007) (Nichols, 2006) where the background

is not so important. In both cases, researchers based

their work on the concept of a single user and a single

device input/output.

This work intends to delve into this part of the

computing science and provide the basis of study

for the creation of user interfaces in environments

with a huge number of heterogeneous users and de-

vices, both in knowledge and needs and capabili-

ties. Therefore, the goal of this work is to de-

ﬁne a model of human-system interaction,advanced

and multi-device. This model should adapt the en-

vironment information, both input and output, in a

proactive way (Tennenhouse, 2008). This adaptation

should be done in a context-oriented way, taking into

account user needs and their proﬁles.

2 SPECIFIC GOALS

The main feature of the environment on which this

work should work is the model of communication that

the Internet of the Future imposes, where the number

of devices and users is uncertain and variable. It is

impossible to know, in advance, the characteristics of

each of these elements. In this scenario is not easy to

determine a model of user-system interaction efﬁcient

309

Vega Barbas M. and R. Velasco J..

INTELLIGENT MULTI-DEVICE USER INTERFACES.

DOI: 10.5220/0003399603090312

In Proceedings of the 1st International Conference on Pervasive and Embedded Computing and Communication Systems (PECCS-2011), pages

309-312

ISBN: 978-989-8425-48-5

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

and correct for every moment.

Therefore, to carry out the work will be necessary

to meet a number of sub-targets to facilitate the at-

tainment of the main goal outlined in the introduc-

tion. These goals can be classiﬁed into the following

points:

1. Deﬁnition of a general interaction model between

environment elements where the Internet of the

Future will deploy.

2. Study of different concepts, design features and

technologies for the development of multi-device

interfaces in order to obtain a semantic model to

tackle the problem consistently.

3. Analysis of the different techniques that enable

proactive systems to establish the relationship be-

tween people and systems, using the more suit-

able Input/Output device at all times. We have

studied the characteristics to be deﬁned within the

context of this particular smart spaces type, being

very important to deﬁne how to create an inter-

face to ensure efﬁcient interaction between users

and the environment.

3 NEW INTERACTION

CONCEPTS FOR INTERNET OF

THE FUTURE

To realize the scope of interaction in the Internet of

the Future is necessary explain two concepts: What

we understand for multi-device interfaces and how

systems can create those interfaces without users su-

pervision.

3.1 Multi-device Interaction

Historically, the development of interfaces has fo-

cused on exploiting the concept of one device - one

user(A. H. Jørgensen, 2008), basically dealing with

environments where the user wants or needs to inter-

act with an application or device. In an environment

consisting of an indeterminate number of I/O devices

and several users with different knowledge levels and

heterogeneous needs, it becomes necessary to move

towards a multi-device interaction to obtain the best

performance.

In the ﬁgure 1 we can observe how the system

transforms a tactile interaction to a gestural interac-

tion by using environment elements such as a web

cam and public screen. For this, the system uses a de-

vice ontology. Each device has a properties list which

the system will use to create the new interface, by

ﬁnding the best I/O device combination.

Figure 1: Transformation from Tactile interaction to multi-

device gestural interaction.

Therefore, we can deﬁne the multi-device inter-

action as the concept where one or more users use

the interaction capabilities of the devices available in

the environment (simultaneous or consecutive) to im-

prove the user experience of offered services. The

idea behind this concept is to facilitate the creation

of similar interfaces using different elements of smart

space, enabling user-system communication in situa-

tions of loss of interaction points.

3.2 Proactive Modeling Techniques for

User Interface and Interaction

Due to that, the goal of this work is to establish

a model and a proactive methodology for the inter-

face creation for Input/Output, it is necessary to de-

termine the technique with which the system must

generate these interfaces. Originally, techniques for

user interface modeling and interaction were devel-

oped to assist software engineers in the implemen-

tation of visual and attractive applications. By us-

ing these techniques they were allowed to raise the

level of abstraction focusing the attention on interac-

tion problems rather than the implementation details

(Szekely, 2006), obtaining products with fewer errors

and higher usability and accessibility degree.

The techniques currently used for developing

user interfaces (XUL, Teresa (Paterno et al., 2008),

UIMLA (UIML, 2009), UsiXML (Limbourg et al.,

2005)) should provide the basis for the development

of new proactive techniques where the system decides

the best interface at any given time. In this way se-

mantics become very important for the description of

the available devices in a smartspace, and so, to deter-

minate the best possible interface combination to cre-

ate the desired interface. Through a proper semantic

description and using appropriate artiﬁcial intelligent

techniques it is possible to deﬁne a user interface that

facilitates user interaction with the system and other-

wise. This has to be done, at least, where the point of

interaction is lost due to communication problems or

infrastructure.

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

310

4 THE CHALLENGE OF

MULTI-DEVICE INTELLIGENT

USER INTERFACES

When someone thinks on functional applications, fo-

cuses attention on the successful execution of an ap-

plication (Nielsen, 1999). Interface usability deter-

mines the quality degree in terms of ease of use, so

this term may be extended to the applications them-

selves. In this sense, the usability of an application

might be deﬁned as the ease of use, i.e. how the ap-

plication can adapt it self to the user needs and the

context in which it is deployed. Therefore, to ensure

that any developed application is capable of being ex-

ecuted in any environment, even in the case you do

not have the interfaces used by default, is a step to-

ward functional usability.

The following points brieﬂy identify the main re-

search areas that need to be improved to solve the

functional usability problems for applications in en-

vironments with a large number of users and devices:

Semantic Standardization. In order to deﬁne each

device and interface uniquely. It is imperative to

implement a standard that allows interoperability

within the Internet of the Future.

Proactive Development. Once the semantic basis is

deﬁned, the system must be able to compose new

interfaces by using available I/O devices. For the

proper development of a new interface is also nec-

essary to automate the speciﬁcation and veriﬁca-

tion of these multi-device interfaces.

Natural Language Processing. Advances in this re-

search ﬁeld should enable the system to perform

the proactive development from a formal descrip-

tion by the users. This should help to increase the

usability since it opens the door for less experi-

enced users.

5 CASE OF STUDY

As part of the ITEA-2 Do-it-Yourself Smart Experi-

ences, a ﬁrst demonstrator of this work has been de-

veloped1. This demonstrator visualizes the power of

this work through a project of creating multi-device

interface for the treatment of public and private infor-

mation. The main idea of this demonstrator is to force

the behavior of a standard output interface to display

information in three different devices depending on

the location and the data privacy.

First, the user, by using a private device, an An-

droid mobile, sends a restaurant menu request to the

server. Then, the server processes the request and re-

trieves the asked information. This information may

be categorized in two types, private and public. Sec-

ond, the server creates two messages, one to return as

response to the mobile device, and another to be put

in a public screen. In this way, the system creates a

interface which is composed with two output devices,

the public screen and the mobile device. A user can

visualize private information in their hands, with the

Android mobile, while the public information is pre-

sented en the screen. The ﬁgure 2 show schematically

the demonstrators behavior.

Figure 2: Multi-device Public/Private Information Inter-

face.

The technology used for the demonstrator devel-

opment is based on the choice of HTTP as a commu-

nication channel and HTML5 as programming lan-

guage for creating applications and REST services.

The information remains stored in a database and is

composed of a user ID and an XML document with

the following structure:

Thus, the output system can identify the sections

of public and private information for processing. This

partition is done by the server, attending a set of rules.

By using a PUSH server-based WebSocket it is possi-

ble to deﬁne two communication channels to send the

information published. Each channels is associated to

a location that is provided to the system in through a

REST request. The system processes information and

returns the private part directly to the device which

made the REST request. The format of this private in-

INTELLIGENT MULTI-DEVICE USER INTERFACES

311

formation is XML, because to make processing easier

in the private device.

6 CONCLUSIONS AND FUTURE

WORKS

The development of the Internet of the Future and

the evolutionary leap of information technologies to-

wards a proactive and ubiquitous model (Tennen-

house, 2008) requires researchers to specify new

forms of interaction, where the user should play a less

intrusive but supervisor role. In addition, applications

need to adapt to new smart environments where they

must deploy and operate.

The work presented in this document aims to de-

ﬁne the basis for the creation of a architectural and se-

mantic model that allows construction of multi-device

interfaces, to improve the interaction between peo-

ple and information systems within intelligent envi-

ronments. By using this model, it will be possible to

deﬁne adaptive applications, in terms of interface, de-

pending of the context where they are deployed and

of the user proﬁle.

Nowadays, our research group, Telematic Ser-

vices Engineering research group (GIST), is working

on the implementation of an interfaces translator. This

research project should be able to create on the ﬂy in-

terfaces from the interaction model which was deﬁned

for a speciﬁc application. The new interface will use

different devices from the environment to obtain, in a

ubiquitous way, the expected functionality.

ACKNOWLEDGEMENTS

This work is framed in the project ITEA-2 Project No

2008005, Do-it-Yourself Smart Experiences, founded

by the Spanish MITC -Avanza- TSI-020400-2009-

124.

REFERENCES

A. H. Jørgensen, B. A. M. (2008). User interface history.

In CHI ’08 extended abstracts on Human factors in

computing systems, pages 2415–2418. ACM.

Buxton, B. (2007). Sketching User Experiences. Morgan

Kaufmann, Washington and Toronto, 2nd edition.

CERP (2009). Internet of things strategic research roadmap.

CERP-IoT group. Retrieved from: http://www.iot-

visitthefuture.eu/.

DiY-SE (2008). Do-it-your-self smart experiences project.

DiY-SE group. Retrieved from: http://dyse.org/.

Krzystof, G. and Daniel, S. W. (2004). Supple: Automati-

cally generating user interfaces. In IUI’04. University

of Washington.

Krzystof, G. and Daniel, S. W. (2007). Automatically gen-

erating user interfaces adapted to users motor and vi-

sion capabilities. In UIST’07. University of Washing-

ton.

Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon,

L., and Lpez-Jaquero, V. (2005). Usixml: A lan-

guage supporting multi-path development of user in-

terfaces. In Engineering Human Computer Interaction

and Interactive Systems, LNCS 3425, pages 200–220.

Springer.

Nichols, J. (2006). Automaticallygenerating user interfaces

for appliances. H-C Interaction Institute, Pittsburgh,

USA.

Nielsen, J. (1999). Designing web usability: The practice of

simplicity. In CHI ’08 extended abstracts on Human

factors in computing systems. New Riders Publishing.

Paterno, F., Santoro, C., Mntyjrvi, J., Mori, G., and San-

sone, S. (2008). Authoring pervasive multimodal user

interfaces. In Int. Journal on Web Engineering and

Technology. 4(2), pages 235–261. ACM.

Saffer, D. (2008). Designing Gestural Interfaces. Morgan

Kaufmann, Washington and Toronto, 2nd edition.

Szekely, P. (2006). Retrospective and challenges for model-

based interface development. In 2nd International

Workshop on Computer-Aided Design of User Inter-

faces. Namur University Press.

Tennenhouse, D. (2008). Proactive computing. In Commu-

nications of the ACM, 43(5), pages 43–50. ACM.

UIML (2009). Uiml deﬁnition. UIML Home page. Re-

trieved from: http://www.uiml.org.

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

312