ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS

INTERACTION IN INTELLIGENT ENVIRONMENTS

Supporting Complex Cooperative Interactions between Humans and Technical

Systems in Real World Scenarios through Cognitive Objects

Matthias Kranz, Andreas Möller and Luis Roalter

Technische Universität München, Arcisstraße 21, 80333 Munich, Germany

Keywords:

Cognition, Intelligent environments, Interaction, Robotics, Objects, Sensors, Actuators, Pervasive computing.

Abstract:

Future intelligent environments will be inhabited by humans, robots and ‘Smart Objects’ and allow for seam-

less interaction beyond the desktop. These environments therefore have to be adaptive, self-organizing, pro-

vide autonomous reasoning and integrate a variety of heterogenous hardware, objects, sensors and actuators –

which goes far beyond merely interconnecting different kinds of technology. In light of the dawn of personal

robotics, these environments should be equally usable and supportive for humans and robots. Manipulation

tasks involving physical objects are at core of the interaction in these environments. This places novel chal-

lenges on the involved ‘Smart Objects’.

We present an approach for supporting robotic systems in the interaction with physical objects while maintain-

ing human usability and functionality by using so-called ‘Cognitive Objects’. We describe our infrastructure

to support developing, simulating, testing and deploying of pervasive computing systems, using ROS (Robot

Operating System) as middleware, and present several application scenarios. The scenarios are not limited

to the robotics domain, but include location-aware services, intelligent environments and mobile interaction

therein. Based on our experience, recommendations for the design of ‘Cognitive Objects’ (CO) and environ-

ments are given, to address the individual strengths of humans and machines and to foster new synergies in

shared human-robot environments.

1 INTRODUCTION

In the last decade, the transition from the classical PC

towards interaction beyond the desktop has begun and

is still ongoing. Our smartphones are our daily com-

panions, our homes become ‘smart’ through embed-

ded sensors and actuators, networking between appli-

ances, and automation. We currently experience the

dawn of personal robotics supporting humans in ev-

eryday life, be it for entertaining purposes or for elder

care scenarios. Unlike highly specialized industrial

robots, the spread towards private households requires

multi-functionality and openness to a broader spec-

trum of application areas. Service robots are intended

for use in highly dynamic real-world environments.

They need to operate electronic devices, interact with

arbitrary objects, and with people. Therefore, robots

need to be aware of their environment and gather in-

formation about it. Today’s sensor- and particularly

vision-based techniques for object detection are not

always sufﬁcient for these scenarios, or have too high

computational demands to work in real-time.

This illustrates a major challenge in the Ubicomp

vision: interaction takes place through objects and ob-

ject manipulation. In robotic environments, objects

therefore need to be designed differently to ﬁt the

needs of both humans and robots. In this paper, we

present an approach for supporting robotic systems in

interaction with physical objects by introducing so-

called Cognitive Objects, maintaining human usabil-

ity at the same time. We demonstrate example scenar-

ios for leveraging interaction between humans, robots

and objects in our intelligent environment, the Cog-

nitive Ofﬁce. We identify strategies for the adaption

of everyday technology for the needs of humans and

technical systems, and provide design recommenda-

tions for objects and environments for more seamless

interaction and applicability for humans and robots.

163

Kranz M., Möller A. and Roalter L..

ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS - Supporting Complex Cooperative

Interactions between Humans and Technical Systems in Real World Scenarios through Cognitive Objects.

DOI: 10.5220/0003323001630172

In Proceedings of the 1st International Conference on Pervasive and Embedded Computing and Communication Systems (PECCS-2011), pages

163-172

ISBN: 978-989-8425-48-5

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

2 COGNITIVE OBJECTS

AND COGNITIVE OFFICE

We present the concept of Cognitive Objects (Möller

et al., 2011) and brieﬂy distinguish them from related

research. Subsequently we introduce the intelligent

environment our Cognitive Objects are embedded in

and summarize the conjoint research goals.

2.1 Deﬁnition of Cognitive Objects

We deﬁne Cognitive Objects as physical artifacts em-

bodied in an interaction which include sensors, ac-

tuators, communication and computation, to equally

support humans and robotic systems in task execu-

tion. This deﬁnition comprises that Cognitive Objects

• are physical, unambigously identiﬁable objects

• are embodied in the environment and the task they

are involved in

• include cognition through sensors, and disclose

information through appropriate actuators

• incorporate communication abilities like Wireless

Sensor Nodes (WSNs)

• proactively and situatedly collaborate with hu-

mans, robots and the environment to assist in task

execution.

Their cooperative nature constitutes a new way to

bridge the gap in interaction between humans and ma-

chines, respecting both human affordances and ma-

chine requirements.

2.2 Related Work on Smart Objects

The idea of augmenting artifacts by digital functional-

ity has been followed with Smart Objects (Kranz and

Schmidt, 2005; Beigl and Gellersen, 2003) or tangible

user interfaces (Ullmer and Ishii, 2000). Their focus

is to move computer interaction away from the WIMP

paradigm (windows, icons, menus and pointer) to-

wards an integration of technology into everyday con-

text and devices. Examples are smart pushpins on no-

ticeboard to stimulate recall (Laerhoven et al., 2002),

a camera-enhanced cabinet to assist humans in object

retrieval (Siio et al., 2003) or edutainment toys (Kranz

et al., 2005). While these concepts focus on humans

and HCI, augmented objects for mixed human-robot

environments must by design equally support humans

and robots.

Smaller Smart Objects are used in the context of

interactive spaces, so-called intelligent environments

(Linner et al., 2010) emerging due to the availability

of embedded computing (Kranz and Schmidt, 2005).

Figure 1: The Cognitive Ofﬁce, our intelligent cognitive en-

vironment in 3D simulation. Physical objects like draw-

ers, windows, doors and plants and corresponding events

are linked to their virtual representations in real-time.

Such sensor-and actuator-augmented rooms can be

the basis for ambient assisted living (Kranz et al.,

2010c) or location-based services. The sensor net-

working platforms used in this context, e.g. Smart-

Its (Holmquist et al., 2004) or Motes

, communicate

wirelessly with other Smart Objects in their environ-

ment dynamically, collect data and perform signal

processing. They consist of standardized hardware

and are not unique like Cognitive Objects, and are not

intended for direct human interaction.

Further information and a detailed discrimination

of Cognitive Objects from other smart object research

can be found in (Möller et al., 2011). Comparisons

to related work will also be made in the respective

locations later in this paper.

2.3 Cognitive Ofﬁce:

An Intelligent Environment

The Cognitive Ofﬁce (Fig. 1) is a live-in lab at our

institute with a multitude of distributed objects, sys-

tems, sensors, and actuators working together. This

‘intelligent environment’ allows for rich and embed-

ded interaction (Kranz et al., 2010b) beyond the desk-

top and provides context-sensitive services to its hu-

man users, the sum of its cooperating systems and ar-

tifacts being more than the individual parts.

The Cognitive Ofﬁce is augmented by various sen-

sors and communication technologies:

• Local sensors and actuators, e.g. for temperature

and light; contact switches for windows, doors

or drawers; PIR sensors for presence detection,

power switches

• Internet-based services (e.g. trafﬁc, weather, ...)

• Cognitive Plants

http://www.xbow.com

http://www.botanicalls.com, http://www.koubachi.com

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

164

An in-depth discussion of this intelligent environ-

ment and its technical basis can be found in (Roalter

et al., 2010).

2.4 Software

The set of software and tools we use for the develop-

ment, simulation, deployment and evaluation of ubiq-

uitous services and intelligent objects in the context

of the Cognitive Ofﬁce consists of three components:

middleware, visualization and simulation.

Middleware. ROS (Robot Operating System) as

underlying middleware interconnects sensors, actua-

tors and all interacting entities in the Cognitive Ofﬁce.

ROS is a meta-operating system initially designed for

robots (ROS, 2010), but equally applicable for an in-

telligent environment. Being a heterogeneous, dis-

tributed sensor-actuator system, such an environment

can be considered as immobile robot (Williams and

Nayak, 1996). The ROS framework builds upon and

abstracts from different underlying systems, contain-

ing packages that abstract from heterogenous hard-

ware, provide low-level control and message-passing

and already implement a wide range of commonly

used functionality, such as controlling a mobile robot.

Visualization. Visualization tools are not only use-

ful as actuators (e.g. for digital door signs, user feed-

back on appliances or status monitoring), but serve

also as a means to visualize received sensor data.

It can be evaluated how a room or an environment

“looks like” for a device that only has certain sensors,

and allows the evaluation of cognition-based systems

in simulated and real intelligent environments. It

thereby does not play a role whether the visualization

output is local or remote.

Simulation. Using simulation tools (e.g. Gazebo

interaction between robots, objects, sensors, and

humans can be re-, but also preconstructed, i.e. sim-

ulated before the actual deployment, using a realistic

physics engine including collision detection and

realistic forces. Building upon standardized object

interchange formats, objects can be designed with

external modeling tools and imported so that the

realistic reconstruction of real-world environments

is possible. Special items, such as robots, e.g. the

Personal Robot 2 (PR2) by Willow Garage

or mobile

phones can be imported as predeﬁned objects.

Using ROS in the context of intelligent environments

has several beneﬁts compared to other middleware.

www.playerstage.sourceforge.net/gazebo/gazebo.html

www.willowgarage.com/pages/pr2/overview

An overview of Ubiquitous Computing middleware

can be found in (Kranz et al., 2007).

Coverage. While many approaches are tailored to a

proposed scenario, ROS covers a wide range of use

cases (see Sec. 3), and is not limited to these. The

support for real robot integration and interaction be-

tween robots and the intelligent environment is a ben-

eﬁt compared to middleware not coming from the

robotics domain.

Reliability. ROS is in a mature state and is continu-

ously further developed, offering high reliability and

frequent updates. Interfaces and APIs are well docu-

mented and users beneﬁt from support and exchange

in the research community.

Applicability. The presented tools support quick

deployment in real-world scenarios, allowing easy ap-

plication implementation on top of abstractions, mod-

els and APIs. Common and frequently used function-

ality is provided ‘out of the box’. A system can be

adapted to different setups with low amount of recon-

ﬁguration. Simulated and real-world sensor data pass

through the same processing chain and cause identi-

cal actuator events, allowing to move from testing to

deployment without code changes.

Openness and Extensibility. Software and code

are open source to enable re-use, allow for code-

inspection and standardization, and have a broad basis

in the community for shared development. The in-

frastructure provides means to add both new soft- and

hardware (such as new algorithms or sensing tech-

nologies). The vividness of the ROS community has

recently been proven again by the integration of MS

Kinect support in ROS only days after the release of

the open source drivers.

2.5 Research Questions and Goals

We pursue the following research questions and goals:

Scenarios. Which scenarios can beneﬁt from the

use of Cognitive Objects that are augmented by com-

munication technology, based on the current state of

research in the areas of robotics and ubiquitous com-

puting? How do such future objects differ from in-

telligent objects as we understand them today, regard-

ing sensors, actuators, distributed cooperative infor-

mation processing, communication and interaction?

Inﬂuence on Interaction. What is the inﬂuence of

such objects on processes and activities for humans,

robots and mixed environments for interaction be-

tween humans, robots and objects, regarding usabil-

ity, utility, subjective and objective quality of interac-

ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -

Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through

Cognitive Objects

165

tion, as well as user transparency and intelligibility?

Relevance for Service Robotics. What is the rel-

evance of such objects for the domain of (service)

robotics? In particular, what are the consequences

regarding the reduction of interaction complexity, al-

gorithms, incertainty and ambiguity of sensor-based

cognition and internal, digital representations of the

real world in dynamic, complex environments? What

contributions can be made to increase the certainty

and reliability of information in robotic perception,

and in which other ways can robotic perception be

augmented, in order to make phenomena perceivable

that today’s robots cannot detect at all, or not reliably

in all desired scenarios?

Abilities. What are characteristic, necessary and

possible abilities and properties of intelligent artifacts

like Cognitive Objects? What is their respective inﬂu-

ence on human and robot interaction with them, and

which possibilities do they open up? How can strate-

gies for the selection of appropriate and adequate in-

formation and communication technologies (ICT) to

be used in such devices and objects be developed?

Development. What are appropriate approaches to

support abstraction and representation in the devel-

opment of such novel objects, both application- and

user-centered? Furthermore, how can at the same

time the integration of electronic hardware and soft-

ware methodically be supported and programming

and customization be facilitated for the end-user?

Security and Privacy. What consequences arise re-

garding the additional complexity of products and a

possible effect on security and privacy of the end

user?

3 APPLICATIONS

We present applications from different ﬁelds inte-

grated and deployed in the context of the previously

described Cognitive Ofﬁce, comprising approaches

from human-object, robot-object and mobile interac-

tion, location- and context-aware services and more

(see Fig. 2).

3.1 Human-computer Interaction and

Human-robot Interaction

We present three constructed prototypes as examples

of Cognitive Objects as deﬁned in Sec. 2.1, and inte-

grated them into the Cognitive Ofﬁce

(two of them

See the Cognitive Ofﬁce and the implemented Cogni-

tive Objects at www.lmt.ei.tum.de/team/kranz/videos.php

ROS Middleware

Visualization

Simulation

Intelligent

Environments

Proactive Services

HCI

Devices and

Smart Objects

Context Awareness

Wireless Sensor Networks

Location Awareness

Positioning and Navigation

Robotics

Simulation and Control

Mobile Interaction

Ubiquitous Control

Figure 2: The robotics middleware ROS in combination

with visualization and simulation tools can serve as a ba-

sis for various applications from different research ﬁelds.

implemented, one simulated).

Cognitive Cup. The Cognitive Cup is a coffee mug

augmented by self-awareness (see Fig. 3). It has an

accelerometer to detect its orientation and senses ﬂuid

level and temperature of its content. The cup helps

robots to track its location and orientation by infrared

LEDs (invisibly for human eyes) built into the seam.

It communicates wirelessly with its environment via

RFID (for identiﬁcation) and ZigBee (for real-time

sensor value transmission). Disclosing this informa-

tion can assist a grasping robot, e.g. to indicate that

the cup has to be handled with care in order not to

be spilled. More sophisticated services become pos-

sible, such as a robot automatically bringing a new

cup when the coffee is cold. The cup still supports

human affordances, as it resembles a normal cup in

shape, size and weight, and can even be put into the

dishwasher (after removing the socket).

Figure 3: The Cognitive Cup supports human affordances

(left) and robot-object interaction through infrared LEDs

(middle). Orientation, ﬂuid level and temperature are trans-

mitted wirelessly, here in 3D visualization (right). See a

video at www.youtube.com/watch?v=mP3ZbPM9TVU.

Load Table. Information about non-augmented ob-

jects and related interaction can be gained through

adding cognition to furniture. Load sensing in a

wooden table provides three information primitives

about objects placed on it: weight, position, and type

of interaction based on the load signal shape over

time (Schmidt et al., 2002). The table becomes a

Cognitive Object disclosing information about other

objects, while the augmentation is entirely invisible

for human eyes. Multiple sensors are networked,

middleware-controlled and allow triangulation of ob-

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

166

ject positions based on measured load values. Addi-

tionally placed objects can be detected by comparing

the capacitive values to the previous state. The distri-

bution of cognition can assist robots in grasping non-

augmented objects which would otherwise be invis-

ible like e.g. glass. The table even detects whether

a glass is full or empty by monitoring the weight

change. A similar approach with capacitive sensors

has been shown in (Wimmer et al., 2007). Currently,

the table is integrated into the simulation only.

Whiteboard Cleaning Robot. Whiteboards are of-

ten used for quick notes. This robotic whiteboard

cleaner autonomously erases the whiteboard, choos-

ing the optimal path to only traverse areas containing

text. At the end, it moves automatically to its charg-

ing station. Before the cleaning procedure, the white-

board’s content is captured and saved as digital image

in order to preserve potentially important information.

The whiteboard is thus enhanced to an interactive sur-

face that autonomously makes information written on

it persistent. The text can be made digitally accessi-

ble and searchable using OCR. Special symbols can

provide additional functionality, e.g. the entire white-

board content can be emailed to a speciﬁed address

by drawing an envelope symbol. While the cleaning

robot in Fig. 4 is an early prototype, in the next iter-

ation the size will be reduced to ﬁt into an everyday,

human-usable whiteboard eraser.

Figure 4: Left: The cleaning robot wiping a whiteboard.

Right: The robot navigates autonomously to ﬁnd the most

efﬁcient route and can be camera-tracked. See a video at

www.youtube.com/watch?v=hZxd-d1dDE4.

3.2 Context and Location Awareness

The localization of objects supports robot-object in-

teraction (e.g. an object being searched can announce

its location) and enable context-sensitive services

(e.g. sending a document to the nearest printer). The

Cognitive Ofﬁce supports both active and passive lo-

calization. Context and location detection using sen-

sors can easily be integrated in the ROS middle-

ware. Passive presence detection (Kranz et al., 2006)

can e.g. be realized using PIR sensors or by infer-

ence from context information such as open doors or

windows with help of a HMM. We use this for au-

tomated HVAC, e.g. switching on the fan not only

temperature-based, but preferably when the worker

leaves the room in order not to distract him by the

noise. Another application would be to disable heat-

ing while the window is open to save energy.

We implement active location detection by

WLAN ﬁngerprinting as indoor positioning system,

using triangulation and similarity search from a

database of location-annotated WLAN access point

signal strengths. Such a system can be the basis

for location-aware objects, up to indoor navigation

with help of a mobile phone. The Cognitive Ofﬁce

and our simulation environment support all stages of

such a system’s development process. In the plan-

ning phase, the 3D simulation of the environment can

visualize the relative signal strength (RSS) of the ex-

isting WLAN base stations including reﬂections etc.,

e.g. using a color-coded map. This can help ﬁnding

positions for measuring ﬁngerprints and adding them

to a database or even to ﬁnd an optimal initial position

of WLAN access points. In a second phase, the posi-

tioning algorithm can be tested in the simulation and

provide valuable insights about its accuracy and relia-

bility before any real deployment, as not only the ﬂoor

plan and walls, but also all furniture and objects of the

real environment are accurately simulated. Changes

can be made easily and tests be rerun, which consti-

tutes an enormous advantage compared to real-world

tests in terms of costs and time.

3.3 Mobile Interaction

Mobile phone interaction approaches with the real

world have been presented, each with individual

drawbacks. Some require the augmentation of objects

with tags (visual or RFID), or special hardware, like

pointing using a laser (Rukzio et al., 2006). Magic-

Phone (Wu et al., 2010) uses the phone’s gyroscope

and accelerometer to detect in which direction it is

pointing has been presented. The phone can then be

used for remote control or interaction with the tar-

geted device, e.g. send slides to a projector or change

TV channels. The position of individual objects in

the room needs however to be known, as the phone’s

orientation is the only clue for detecting which device

it is pointing at. For the development and testing of

such a system, the whole environment would have to

set up, requiring physical space, time and costs.

Our approach using ROS and simulation tools

could perfectly support such a setup in simulation and

testing. The MagicPhone scenario has been realized

in simulation in our system to highlight the potential

of the proposed toolchain. The model of the mobile

ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -

Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through

Cognitive Objects

167

phone can be imported in simulation tool and inte-

grated into the virtual environment, which is an exact

model of the real-world rooms in which the system is

to be tested. Coming from the robotics area, ROS con-

tains methods for remotely controlling robots in an

environment, showing the robot’s exact ﬁeld of view

in the simulation. These tools can similarly be used

for the simulation of a mobile phone carried by a per-

son. Pointing interactions are simulated by control-

ling the phone model using the PC keyboard, a multi-

axis controller or a MS Kinect camera. The object(s)

targeted by the phone can easily be detected by view-

ing the phone’s simulated camera image in the simu-

lation.

3.4 Extending Intelligent Environments

Intelligent environments or ‘smart spaces’ can be de-

ﬁned as multi-user, multi-device, dynamic interaction

environments that enhance a physical space by virtual

services (Johanson et al., 2002). The aspect of be-

ing ‘dynamic’ comprises the smart space’s extension

over the borders of an augmented room to wherever

the user is, e.g. on the way from and to the ofﬁce.

The Cognitive Ofﬁce extends its functionality out

of the physical ofﬁce space by receiving a notiﬁcation

from the user’s smartphone when he approaches the

campus (detected via GPS or the WLAN SSID). The

ofﬁce PC is thereupon booted via Wake-on-LAN, im-

portant appointments are shown on the calendar and

the email application is launched, ready to use when

the user reaches the ofﬁce. The Cognitive Ofﬁce also

takes care of a pleasant trip home: A trafﬁc service

knowing the way home proactively searches for traf-

ﬁc news on that route. In case of a trafﬁc jam or ac-

cident, this news is reported even before the user is

getting in his car. If historically trafﬁc jams occurred

frequently at a certain time (e.g. the rush hours), the

system proactively suggests the right time to leave the

ofﬁce by self-learning.

3.5 Simulating Robots and Humans

Coming from the robotics domain, the ROS middle-

ware is perfectly suitable for monitoring, controlling

and evaluating interaction between robots and hu-

mans, objects and the environment. ROS incorporates

various methods for robotic control and interaction,

as well as drivers for components like vision systems.

We use ROS to control the PR2 in the Cognitive Ofﬁce

(as seen in Fig. 5 in the 3D simulation).

Figure 5: Visualization of a PR2 robot’s hand camera views.

Due to the support of complex motion sequences

and the integrated physics engine, a bio-mechanical

human model can be integrated for simulating inter-

action with (intelligent) artifacts, robots or any other

objects in the virtual environment. ROS supports

complete control of arms, head and other parts of

the (robot) body, which can be adapted for a human

model. Different types of interaction, e.g. pressing

buttons, grasping objects, opening doors etc. can be

modeled and tested. Models can be controlled by

keyboard or another controller, e.g. a joystick. It is

imaginable to steer a human model through an ac-

curate model of a 3D simulation and use it to eval-

uate location-aware systems, e.g. presence detection

for proactive context-based services.

The visualization tool allows to monitor the

robot’s camera views as it moves in real time and thus

‘see the world through the robot’s eyes’. It is also pos-

sible to monitor the sensory input such as the robot’s

laser scanner, both from the real world and the simula-

tion. This allows to simulate a robot exploring an un-

known environment and to record the data it captures

before any deployment in the real world, which con-

stitutes valuable information for evaluating whether

e.g a service robot would get along in a domestic en-

vironment.

3.6 Interconnection

All examples just described are not individual appli-

cations put together in one room, but share the same

middleware and can communicate with each other.

The ROS publish/subscribe architecture lets arbitrary

components use any data and service available to the

middleware, allowing for a new stage of smart ser-

vices, and interaction between objects, robots and hu-

mans.

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

168

4 DESIGN GUIDELINES

We have presented our research goals to leverage

human-robot and robot-object interaction in the Cog-

nitive Ofﬁce with help of Cognitive Objects. We

have presented examples from various contexts that

we have successfully developed, designed and tested

with help of this environment. In the remainder of this

paper, we draw ﬁrst conclusions from our experiences

and deduce guidelines for the design of intelligent ar-

tifacts and environments that support interaction be-

tween humans, objects and robots. We begin with

comparing the abilities and characteristics of human

and machines, and afterwards give recommendations

for how objects and environments could be designed

to play out the strengths and compensate the weak-

nesses of humans and computerized systems for more

effective interaction, collaboration and cooperation.

4.1 Human-machine Comparison

As a basis for the later recommendations, the differ-

ences between humans and machines in terms of their

unequal abilities are described.

4.1.1 Human Abilities and Machine Abilities

The following classiﬁcations (Chapanis et al., 1951)

have been made in the context of aviation in order

to choose whether functions can be controlled by ma-

chines or are better operated by humans. These obser-

vations also apply for today’s computerized systems

and support function allocation in intelligent environ-

ments. Humans surpass machines in their ability to

• detect small amount of visual and acoustic energy

• perceive patterns of light or sound

• improvise and use ﬂexible procedures

• store very large amounts of information for long

periods and to recall relevant facts at the appropri-

ate time

• reason inductively

• exercise judgment.

Machines appear to surpass humans with respect to

the ability to

• respond quickly to control signals, and to apply

great force smoothly and precisely

• perform repetitive, routine tasks

• to store information brieﬂy and then to erase it

completely

• to reason deductively, including computational

ability

• handle complex operations, i.e. to do many differ-

ent things at once.

4.1.2 People versus Machine-centered Views

The above abilities appear differently depending on

the point of view. From a machine-centered view,

people are inexact, often act illogically or emotion-

ally. Machines, by contrast, are always precise, or-

derly and logical. However, from a people-centered

view, machines lack essential abilities, such as cre-

ativity, imagination or adaptability, are ‘dumb’, while

humans are resourceful, creative and attentive to

changes. The fact that the strengths of one party are

weaknesses of the other (see Fig. 6) needs to be con-

sidered in the design of objects and environments con-

jointly used by humans and machines.

People are

Machines are

Machine-

centered

view

• vague

• disorganized

• distractible

• emotional

• illogical

• precise

• orderly

• undistractible

• unemotional

• logical

People-

centered

view

• creative

• compliant

• attentive to changes

• resourceful

• able to make ﬂexible

decisions based on context

• dumb

• rigid

• insensitive to change

• unimaginative

• constrained to make consistent

decisions

Figure 6: Humans and machines have different abilities

which are – depending on a people-centered or machine-

centered view – strengths or weaknesses. Intelligent en-

vironments should both humans and computers support in

their strengths, and compensate their weaknesses. Tabular

representation according to (Norman, 1993).

4.2 Recommendations

Based on human and machine abilities and our experi-

ence in the Cognitive Ofﬁce, we present initial recom-

mendations for the design of intelligent environments

and Smart Objects. We identify object and environ-

ment properties, and classify them according to their

importance for humans and robotic systems, as sum-

marized in Fig. 7. Objects and environments intended

for a conjoint use by humans and robots should re-

spect the recommendations from both categories.

4.2.1 Important Object Properties

Smart Objects should respect the following properties

to support interaction with robotic systems:

Visibility. The ﬁrst precondition for interaction is

the object’s visibility to a robot, being a challenging

task for computer vision systems when the object is

transparent or has glossy and shiny surfaces. Small

objects might be hardly distinguishable. Low-light

ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -

Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through

Cognitive Objects

169

Object Properties

Environment Properties

Important

for robots

• Visibility

• Identiﬁability

• Physical Interactability

• Cooperation

• Location Awareness

• Context/

Situation Awareness

• Accessibility

Important

for humans

• Usability

• Convenience

• Comfort

• Optionality of Technical Systems

• Well-Being and Style

• Affordance

Figure 7: Objects and environments for shared robot and

human use should support speciﬁc aspects, some of them

being especially important for humans or for computerized

systems.

environments, shadows or occlusions (e.g. because of

other objects in the line of sight or the robotic arm in

the way) can aggravate the problem. Objects can sup-

port such situations by ‘I am here’ self-disclosement,

e.g. by built-in RFID tags or infrared LEDs invisible

for humans.

Identiﬁability. Beyond being visible, objects need

to be identiﬁable by robots as speciﬁc object out of

many similar ones. This could likewise be realized

using RFID tags, as demonstrated in our Cognitive

Cup. Invisible technology like RFID helps maintain-

ing the object’s initial appearance, while visual tags

like QR codes can also be useful for humans, serving

as an indicator for embedded functionality.

Physical Interactability. Objects need to support

physical interaction with robots, e.g. be graspable by

robotic hands. Not only the shape of the object and

possible handles need to be formed accordingly, the

robot needs to know how to grasp it (e.g. a cup ﬁlled

with hot coffee needs to be grasped differently than an

empty cup, which can also be picked up on the handle

with the open side hanging down). Cognitive Objects

can assist interaction by disclosing information about

their internal state, e.g. whether a cup is empty or not.

Moreover, infrared LEDs can, invisibly for humans,

support pose estimation for a robot, which eases the

task of targeting it. The object can be adapted to the

needs of robot interaction, as long as it does not neg-

atively affect human interaction. The surface of our

Cognitive Cup is e.g. rougher than a porcelain mug,

in order to facilitate grasping by a robot hand.

Cooperation. Objects should cooperate in interac-

tion with robotic systems, instead of leaving cognition

and computation all to the robot. The distribution task

execution conforming the Ubiquitous Computing vi-

sion (Weiser, 1991) is more efﬁcient in terms of com-

putational power and more effective. Cognitive Ob-

jects follow this paradigm (see (Möller et al., 2011)

for more detailed information).

Affordance. The term of affordance is placed at the

intersection of humans and machines in Fig. 7, as it is

equally important for both of them. In usability and

UI design, affordance denotes the ability of an object

to explain itself (Norman, 1990), i.e. to make intelli-

gible to an interacting person what it is for and how

to use it. Humans know – based on their experience

– how to use e.g. a teapot and where to grasp it when

hot. Robot-usable everyday objects need special af-

fordances, which have been subject to research e.g. in

(Duchon et al., 1994) and (Fitzpatrick et al., 2003).

Affordances for robots can be realized by disclosing

their purpose and usage instructions in a machine-

understandable way. Cognitive Objects incorporating

sensors, actuators, computation and communication

are a means to provide this semantical knowledge.

The augmentation for robot usability must not go

along with replacement, but coexistence of human

affordances. Just as human affordances are ‘invisible’

for robots, parts relevant only for technical systems

(like e.g. antennas) should be hidden from human

eyes.

For human usage, Smart Objects should respect

Usability. To be practical and simple to use for hu-

mans, interaction design, cognitive psychology and

cultural factors should be considered in the design

progress. Consider an ‘intelligent’ cup signaling the

temperature of its content. While the cup internally

possibly works with a numeric temperature scale, the

temperature might be glimpsed more intuitively by a

person if it is translated into a color scale (red = hot,

blue = cold), instead of just showing the numeric tem-

perature value on a display.

Convenience. Augmented objects must not re-

duce convenience compared to conventional, non-

augmented objects. When e.g. the battery is drained,

the object should still offer its non-technical function-

ality, instead of not being usable at all.

4.2.2 Important Properties for Environments

For an intelligent environment being used by robotic

systems, the following factors are of importance:

Location Awareness. For a human, it is self-

evident to know whether he is in the kitchen or in the

living room, while for a robot, it isn’t. Intelligent en-

vironments suitable for robots thus should support lo-

cation awareness for computerized entities. They can

e.g. be equipped with multiple WLAN base stations

to enable WLAN ﬁngerprinting, or offer positioning

using DECT (Kranz et al., 2010a), especially for Cog-

nitive Objects.

Context Awareness. Intelligent environments aim

at facilitating people’s everyday life by context-aware

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

170

services and applications. The environment can sup-

port machines in providing this context awareness,

e.g. by providing the respective sensors to detect dif-

ferent sorts of events, such as presence by motion sen-

sors, opening or closing of windows, doors and draw-

ers by contact switches, etc.

Accessibility. In a mixed human-robot environment

all rooms and areas must be accessible by robotic

locomotion systems. While some robots are able to

use stairs, for most wheel-based robots, stairways

are an insuperable obstacle. The width of doorways,

the space between furniture or missing space to turn

around can likewise limit robots to access a desired

place. The height of desks, cupboards or switches

needs to match the robot’s ability to move and stretch

in order to enable interaction. As vision-based

recognition systems perform worse in recognizing

objects in bad lightning conditions, the installation of

additional light sources could produce relief.

Important for humans are the following factors in the

design of an intelligent environment:

Comfort. Living spaces and work places should as-

sist human users by offering a practical environment,

i.e. all needed objects in a certain situation are placed

within reach, the furnishings and facilities ﬁt the spe-

ciﬁc needs, etc. For additional comfort, context-

aware services and proactive functionality simplify

repeating tasks and remove complexity. Technical

systems should take over activities according to their

strengths (see Sec. 4.1.1), such as routine tasks, sim-

ple control actions like switching on the light on ac-

tivity, but leave complex or creative tasks to humans.

Optionality of Technical Systems. Technical sys-

tems should not entirely replace conventional ways of

task completion, following the principle of ‘augmen-

tation instead of automation’. Proactive behavior of

an intelligent home should not manifest in patroniz-

ing inhabitants, but rather in offers and suggestions

they can adopt or silently ignore. The ability to over-

ride the system’s ‘intelligence’ needs to be given at

every time.

Well-being and Style. People want to feel com-

fortable in their environment, recognizable by the

amounts of money spent for attractiveness and de-

sign. Besides functionality, the emotional relationship

to objects and the so-called ‘visceral design’ (Nor-

man, 2003) become increasingly important. A chair is

no longer an object for comfortable sitting but some-

times a piece of art. A similar trend is observable in

electronics, with regard to thoroughly and artistically

designed phones, computers or home entertainment

components, using expensive materials like glass and

aluminum. Environments designed for machines and

humans cannot neglect this requirement any more.

Although being suitable for robot use, such environ-

ments need to satisfy the human wishes of a pleasur-

able, nicely designed home.

5 CONCLUSIONS

We have presented an approach for supporting robotic

systems in interaction with physical objects by Cog-

nitive Objects, with which we have opened a com-

pletely new research ﬁeld at the intersection of per-

vasive computing and personal robotics. Cognitive

Objects are real-world artifacts embodied in an in-

teraction that include sensors, actuators, communica-

tion, and computation, cooperating with humans and

robots to support task execution. Today’s robotic vi-

sion systems, and overall interaction capabilities, are

not sufﬁcient for arbitrary object recognition, hin-

dering a full integration of personal robotics in real-

world environments. By cooperatively disclosing in-

formation about themselves, Cognitive Objects facil-

itate the recognition of and interaction with any kind

of object by robots and thus are a step towards more

comprehensive robot-object interaction. At the same

time, Cognitive Objects are not singularly designed

for machine-machine interaction, but remain equally

usable by humans, maintaining usability and affor-

dances.

We demonstrated the potential for shared object

usage by humans and robots in our Cognitive Ofﬁce,

an intelligent environment using a robotic middle-

ware, and showed the potential for supporting mobile

interaction, context and location awareness.

We have given initial recommendations for the de-

sign of objects and environments with regard to a

shared usage by robots and humans. Cognitive Ob-

jects can sustainably support and stimulate research

in personal and robot interaction, as well as ubiqui-

tous computing, as they provide valuable ground truth

data to robots and facilitate interaction for both hu-

mans and robots. In the future, we plan to integrate

further Cognitive Objects into our intelligent environ-

ment, focusing on supporting ambient assisted living

and personal robotics scenarios and their evaluation

in complex interactions.

Resources. Source code, documentation and data

sets of the presented systems and environment are

shared via https://vmi.lmt.ei.tum.de/ros/.

ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -

Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through

Cognitive Objects

171

ACKNOWLEDGEMENTS

This work has been funded in parts from the German

DFG funded Cluster of Excellence ‘CoTeSys – Cog-

nition for Technical Systems’ and the German BMBF

funded project ‘GEWOS – Gesund wohnen mit Stil’.

REFERENCES

Beigl, M. and Gellersen, H. (2003). Smart-its: An embed-

ded platform for smart objects. In In Proc. Smart Ob-

jects Conference (SOC 2003, pages 15–17.

Chapanis, A., Frick, F. C., Garner, W. R., Gebhard, J. W.,

Grether, W. F., Henneman, R. H., Kappauf, W. E.,

Newman, E. B., and Williams, A. C. J. (1951). Hu-

man engineering for an effective air-navigation and

trafﬁc control system. National Academy of Sciences

Archives.

Duchon, A. P., Warren, W. H., and Kaelbling, L. P. (1994).

Ecological robotics. Adaptive Behavior, 6.

Fitzpatrick, P., Metta, G., Natale, L., Rao, S., Sandini, G.,

and S, G. (2003). Learning about objects through ac-

tion - initial steps towards artiﬁcial cognition. In In

Proc. ICRA, pages 3140–3145.

Holmquist, L., Gellersen, H., Kortuem, G., Antifakos, S.,

Michahelles, F., Schiele, B., Beigl, M., and Maze, R.

(2004). Building intelligent environments with Smart-

Its. Computer Graphics and Applications, IEEE,

24(1):56–64.

Johanson, B., Fox, A., and Winograd, T. (2002). The inter-

active workspaces project: experiences with ubiqui-

tous computing rooms. Pervasive Computing, IEEE,

1(2):67–74.

Kranz, M., Fischer, C., and Schmidt, A. (2010a). A com-

parative study of DECT and WLAN signals for indoor

localization. In PerCom 2010, pages 235 –243. IEEE.

Kranz, M., Holleis, P., and Schmidt, A. (2006). Ubiquitous

Presence Systems. In SAC ’06: Proceedings of the

2006 ACM symposium on Applied computing, pages

1902–1909. ACM.

Kranz, M., Holleis, P., and Schmidt, A. (2010b). Embedded

Interaction: Interacting with the Internet of Things.

IEEE Internet Computing, 14:46–53.

Kranz, M., Linner, A., Ellmann, B., Bittner, A., Bock, T.,

and Roalter, L. (2010c). Robotic Service Cores for

Ambient Assisted Living. Pervasive Health 2010.

Kranz, M. and Schmidt, A. (2005). Prototyping Smart

Objects for Ubiquitous Computing. In Workshop on

Smart Object Systems, UbiComp.

Kranz, M., Schmidt, A., Rusu, R., Maldonado, A., Beetz,

M., Hornler, B., and Rigoll, G. (2007). Sensing

Technologies and the Player-Middleware for Context-

Awareness in Kitchen Environments. In Proc. of INSS,

pages 179 –186.

Kranz, M., Schmidt, D., Holleis, P., and Schmidt, A. (2005).

A Display Cube as a Tangible User Interface. Adj. In

Proc. of UbiComp 2005. Springer.

Laerhoven, K. V., Schmidt, A., and Gellersen, H.-W.

(2002). Pin&play: Networking objects through pins.

In UbiComp ’02, pages 219–228. Springer-Verlag.

Linner, T., Kranz, M., Roalter, L., and Bock, T. (2010).

Compacted and industrially customizable ambient in-

telligent service units: Typology, examples and per-

formance. In 2010 Sixth International Conference on

Intelligent Environments, pages 295–300.

Möller, A., Roalter, L., and Kranz, M. (2011). Cog-

nitive Objects for Human-Computer Interaction and

Human-Robot Interaction. In 6th ACM/IEEE Interna-

tional Conference on Human-Robot Interaction.

Norman, D. A. (1990). The design of everyday things. Dou-

bleday.

Norman, D. A. (1993). Things that make us smart: de-

fending human attributes in the age of the machine.

Addison-Wesley Pub. Co.

Norman, D. A. (2003). Emotional Design: Why We Love

(or Hate) Everyday Things. Basic Books.

Roalter, L., Kranz, M., and Möller, A. (2010). A middle-

ware for intelligent environments and the internet of

things. In Ubiquitous Intelligence and Computing,

volume 6406 of Lecture Notes in Computer Science,

pages 267–281. Springer Berlin / Heidelberg.

ROS (2010). An open-source, meta-operating system

for your robot, robot operating system middleware.

http://www.ros.org.

Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P.,

Schmidt, A., and Chin, J. (2006). An experimental

comparison of physical mobile interaction techniques:

Touching, pointing and scanning. In Dourish, P. and

Friday, A., editors, UbiComp 2006: Ubiquitous Com-

puting, volume 4206 of Lecture Notes in Computer

Science, pages 87–104. Springer Berlin / Heidelberg.

Schmidt, A., Strohbach, M., van Laerhoven, K., Friday,

A., and Gellersen, H.-W. (2002). Context acquisition

based on load sensing. In Borriello, G. and Holmquist,

L., editors, UbiComp 2002: Ubiquitous Computing,

volume 2498 of Lecture Notes in Computer Science,

pages 161–192. Springer Berlin / Heidelberg.

Siio, I., Rowan, J., and Mynatt, E. (2003). Finding objects

in "strata drawer". In CHI ’03: CHI ’03 extended ab-

stracts on Human factors in computing systems, pages

982–983. ACM.

Ullmer, B. and Ishii, H. (2000). Emerging Frameworks for

Tangible User Interfaces. IBM Syst. J., 39(3-4):915–

931.

Weiser, M. (1991). The Computer For The 21st Century.

Scientiﬁc American, 265(3):66–75.

Williams, B. C. and Nayak, P. P. (1996). Immobile Robots

AI in the New Millennium. AI Magazine, 17(3):16–

35.

Wimmer, R., Kranz, M., Boring, S., and Schmidt, A.

(2007). Captable and capshelf - unobtrusive activity

recognition using networked capacitive sensors. In

Proc. Fourth International Conference on Networked

Sensing Systems INSS ’07, pages 85–88.

Wu, J., Pan, G., Zhang, D., Li, S., and Wu, Z. (2010). Mag-

icphone: pointing & interacting. In Ubicomp ’10:

Proceedings of the 12th ACM international confer-

ence adjunct papers on Ubiquitous computing, pages

451–452. ACM.

PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems

172