ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS
INTERACTION IN INTELLIGENT ENVIRONMENTS
Supporting Complex Cooperative Interactions between Humans and Technical
Systems in Real World Scenarios through Cognitive Objects
Matthias Kranz, Andreas Möller and Luis Roalter
Technische Universität München, Arcisstraße 21, 80333 Munich, Germany
Keywords:
Cognition, Intelligent environments, Interaction, Robotics, Objects, Sensors, Actuators, Pervasive computing.
Abstract:
Future intelligent environments will be inhabited by humans, robots and ‘Smart Objects’ and allow for seam-
less interaction beyond the desktop. These environments therefore have to be adaptive, self-organizing, pro-
vide autonomous reasoning and integrate a variety of heterogenous hardware, objects, sensors and actuators –
which goes far beyond merely interconnecting different kinds of technology. In light of the dawn of personal
robotics, these environments should be equally usable and supportive for humans and robots. Manipulation
tasks involving physical objects are at core of the interaction in these environments. This places novel chal-
lenges on the involved ‘Smart Objects’.
We present an approach for supporting robotic systems in the interaction with physical objects while maintain-
ing human usability and functionality by using so-called ‘Cognitive Objects’. We describe our infrastructure
to support developing, simulating, testing and deploying of pervasive computing systems, using ROS (Robot
Operating System) as middleware, and present several application scenarios. The scenarios are not limited
to the robotics domain, but include location-aware services, intelligent environments and mobile interaction
therein. Based on our experience, recommendations for the design of ‘Cognitive Objects’ (CO) and environ-
ments are given, to address the individual strengths of humans and machines and to foster new synergies in
shared human-robot environments.
1 INTRODUCTION
In the last decade, the transition from the classical PC
towards interaction beyond the desktop has begun and
is still ongoing. Our smartphones are our daily com-
panions, our homes become ‘smart’ through embed-
ded sensors and actuators, networking between appli-
ances, and automation. We currently experience the
dawn of personal robotics supporting humans in ev-
eryday life, be it for entertaining purposes or for elder
care scenarios. Unlike highly specialized industrial
robots, the spread towards private households requires
multi-functionality and openness to a broader spec-
trum of application areas. Service robots are intended
for use in highly dynamic real-world environments.
They need to operate electronic devices, interact with
arbitrary objects, and with people. Therefore, robots
need to be aware of their environment and gather in-
formation about it. Today’s sensor- and particularly
vision-based techniques for object detection are not
always sufficient for these scenarios, or have too high
computational demands to work in real-time.
This illustrates a major challenge in the Ubicomp
vision: interaction takes place through objects and ob-
ject manipulation. In robotic environments, objects
therefore need to be designed differently to fit the
needs of both humans and robots. In this paper, we
present an approach for supporting robotic systems in
interaction with physical objects by introducing so-
called Cognitive Objects, maintaining human usabil-
ity at the same time. We demonstrate example scenar-
ios for leveraging interaction between humans, robots
and objects in our intelligent environment, the Cog-
nitive Office. We identify strategies for the adaption
of everyday technology for the needs of humans and
technical systems, and provide design recommenda-
tions for objects and environments for more seamless
interaction and applicability for humans and robots.
163
Kranz M., Möller A. and Roalter L..
ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS - Supporting Complex Cooperative
Interactions between Humans and Technical Systems in Real World Scenarios through Cognitive Objects.
DOI: 10.5220/0003323001630172
In Proceedings of the 1st International Conference on Pervasive and Embedded Computing and Communication Systems (PECCS-2011), pages
163-172
ISBN: 978-989-8425-48-5
Copyright
c
2011 SCITEPRESS (Science and Technology Publications, Lda.)
2 COGNITIVE OBJECTS
AND COGNITIVE OFFICE
We present the concept of Cognitive Objects (Möller
et al., 2011) and briefly distinguish them from related
research. Subsequently we introduce the intelligent
environment our Cognitive Objects are embedded in
and summarize the conjoint research goals.
2.1 Definition of Cognitive Objects
We define Cognitive Objects as physical artifacts em-
bodied in an interaction which include sensors, ac-
tuators, communication and computation, to equally
support humans and robotic systems in task execu-
tion. This definition comprises that Cognitive Objects
are physical, unambigously identifiable objects
are embodied in the environment and the task they
are involved in
include cognition through sensors, and disclose
information through appropriate actuators
incorporate communication abilities like Wireless
Sensor Nodes (WSNs)
proactively and situatedly collaborate with hu-
mans, robots and the environment to assist in task
execution.
Their cooperative nature constitutes a new way to
bridge the gap in interaction between humans and ma-
chines, respecting both human affordances and ma-
chine requirements.
2.2 Related Work on Smart Objects
The idea of augmenting artifacts by digital functional-
ity has been followed with Smart Objects (Kranz and
Schmidt, 2005; Beigl and Gellersen, 2003) or tangible
user interfaces (Ullmer and Ishii, 2000). Their focus
is to move computer interaction away from the WIMP
paradigm (windows, icons, menus and pointer) to-
wards an integration of technology into everyday con-
text and devices. Examples are smart pushpins on no-
ticeboard to stimulate recall (Laerhoven et al., 2002),
a camera-enhanced cabinet to assist humans in object
retrieval (Siio et al., 2003) or edutainment toys (Kranz
et al., 2005). While these concepts focus on humans
and HCI, augmented objects for mixed human-robot
environments must by design equally support humans
and robots.
Smaller Smart Objects are used in the context of
interactive spaces, so-called intelligent environments
(Linner et al., 2010) emerging due to the availability
of embedded computing (Kranz and Schmidt, 2005).
Figure 1: The Cognitive Office, our intelligent cognitive en-
vironment in 3D simulation. Physical objects like draw-
ers, windows, doors and plants and corresponding events
are linked to their virtual representations in real-time.
Such sensor-and actuator-augmented rooms can be
the basis for ambient assisted living (Kranz et al.,
2010c) or location-based services. The sensor net-
working platforms used in this context, e.g. Smart-
Its (Holmquist et al., 2004) or Motes
1
, communicate
wirelessly with other Smart Objects in their environ-
ment dynamically, collect data and perform signal
processing. They consist of standardized hardware
and are not unique like Cognitive Objects, and are not
intended for direct human interaction.
Further information and a detailed discrimination
of Cognitive Objects from other smart object research
can be found in (Möller et al., 2011). Comparisons
to related work will also be made in the respective
locations later in this paper.
2.3 Cognitive Office:
An Intelligent Environment
The Cognitive Office (Fig. 1) is a live-in lab at our
institute with a multitude of distributed objects, sys-
tems, sensors, and actuators working together. This
‘intelligent environment’ allows for rich and embed-
ded interaction (Kranz et al., 2010b) beyond the desk-
top and provides context-sensitive services to its hu-
man users, the sum of its cooperating systems and ar-
tifacts being more than the individual parts.
The Cognitive Office is augmented by various sen-
sors and communication technologies:
Local sensors and actuators, e.g. for temperature
and light; contact switches for windows, doors
or drawers; PIR sensors for presence detection,
power switches
Internet-based services (e.g. traffic, weather, ...)
Cognitive Plants
2
1
http://www.xbow.com
2
http://www.botanicalls.com, http://www.koubachi.com
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
164
An in-depth discussion of this intelligent environ-
ment and its technical basis can be found in (Roalter
et al., 2010).
2.4 Software
The set of software and tools we use for the develop-
ment, simulation, deployment and evaluation of ubiq-
uitous services and intelligent objects in the context
of the Cognitive Office consists of three components:
middleware, visualization and simulation.
Middleware. ROS (Robot Operating System) as
underlying middleware interconnects sensors, actua-
tors and all interacting entities in the Cognitive Office.
ROS is a meta-operating system initially designed for
robots (ROS, 2010), but equally applicable for an in-
telligent environment. Being a heterogeneous, dis-
tributed sensor-actuator system, such an environment
can be considered as immobile robot (Williams and
Nayak, 1996). The ROS framework builds upon and
abstracts from different underlying systems, contain-
ing packages that abstract from heterogenous hard-
ware, provide low-level control and message-passing
and already implement a wide range of commonly
used functionality, such as controlling a mobile robot.
Visualization. Visualization tools are not only use-
ful as actuators (e.g. for digital door signs, user feed-
back on appliances or status monitoring), but serve
also as a means to visualize received sensor data.
It can be evaluated how a room or an environment
“looks like” for a device that only has certain sensors,
and allows the evaluation of cognition-based systems
in simulated and real intelligent environments. It
thereby does not play a role whether the visualization
output is local or remote.
Simulation. Using simulation tools (e.g. Gazebo
3
),
interaction between robots, objects, sensors, and
humans can be re-, but also preconstructed, i.e. sim-
ulated before the actual deployment, using a realistic
physics engine including collision detection and
realistic forces. Building upon standardized object
interchange formats, objects can be designed with
external modeling tools and imported so that the
realistic reconstruction of real-world environments
is possible. Special items, such as robots, e.g. the
Personal Robot 2 (PR2) by Willow Garage
4
or mobile
phones can be imported as predefined objects.
Using ROS in the context of intelligent environments
has several benefits compared to other middleware.
3
www.playerstage.sourceforge.net/gazebo/gazebo.html
4
www.willowgarage.com/pages/pr2/overview
An overview of Ubiquitous Computing middleware
can be found in (Kranz et al., 2007).
Coverage. While many approaches are tailored to a
proposed scenario, ROS covers a wide range of use
cases (see Sec. 3), and is not limited to these. The
support for real robot integration and interaction be-
tween robots and the intelligent environment is a ben-
efit compared to middleware not coming from the
robotics domain.
Reliability. ROS is in a mature state and is continu-
ously further developed, offering high reliability and
frequent updates. Interfaces and APIs are well docu-
mented and users benefit from support and exchange
in the research community.
Applicability. The presented tools support quick
deployment in real-world scenarios, allowing easy ap-
plication implementation on top of abstractions, mod-
els and APIs. Common and frequently used function-
ality is provided ‘out of the box’. A system can be
adapted to different setups with low amount of recon-
figuration. Simulated and real-world sensor data pass
through the same processing chain and cause identi-
cal actuator events, allowing to move from testing to
deployment without code changes.
Openness and Extensibility. Software and code
are open source to enable re-use, allow for code-
inspection and standardization, and have a broad basis
in the community for shared development. The in-
frastructure provides means to add both new soft- and
hardware (such as new algorithms or sensing tech-
nologies). The vividness of the ROS community has
recently been proven again by the integration of MS
Kinect support in ROS only days after the release of
the open source drivers.
2.5 Research Questions and Goals
We pursue the following research questions and goals:
Scenarios. Which scenarios can benefit from the
use of Cognitive Objects that are augmented by com-
munication technology, based on the current state of
research in the areas of robotics and ubiquitous com-
puting? How do such future objects differ from in-
telligent objects as we understand them today, regard-
ing sensors, actuators, distributed cooperative infor-
mation processing, communication and interaction?
Influence on Interaction. What is the influence of
such objects on processes and activities for humans,
robots and mixed environments for interaction be-
tween humans, robots and objects, regarding usabil-
ity, utility, subjective and objective quality of interac-
ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -
Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through
Cognitive Objects
165
tion, as well as user transparency and intelligibility?
Relevance for Service Robotics. What is the rel-
evance of such objects for the domain of (service)
robotics? In particular, what are the consequences
regarding the reduction of interaction complexity, al-
gorithms, incertainty and ambiguity of sensor-based
cognition and internal, digital representations of the
real world in dynamic, complex environments? What
contributions can be made to increase the certainty
and reliability of information in robotic perception,
and in which other ways can robotic perception be
augmented, in order to make phenomena perceivable
that today’s robots cannot detect at all, or not reliably
in all desired scenarios?
Abilities. What are characteristic, necessary and
possible abilities and properties of intelligent artifacts
like Cognitive Objects? What is their respective influ-
ence on human and robot interaction with them, and
which possibilities do they open up? How can strate-
gies for the selection of appropriate and adequate in-
formation and communication technologies (ICT) to
be used in such devices and objects be developed?
Development. What are appropriate approaches to
support abstraction and representation in the devel-
opment of such novel objects, both application- and
user-centered? Furthermore, how can at the same
time the integration of electronic hardware and soft-
ware methodically be supported and programming
and customization be facilitated for the end-user?
Security and Privacy. What consequences arise re-
garding the additional complexity of products and a
possible effect on security and privacy of the end
user?
3 APPLICATIONS
We present applications from different fields inte-
grated and deployed in the context of the previously
described Cognitive Office, comprising approaches
from human-object, robot-object and mobile interac-
tion, location- and context-aware services and more
(see Fig. 2).
3.1 Human-computer Interaction and
Human-robot Interaction
We present three constructed prototypes as examples
of Cognitive Objects as defined in Sec. 2.1, and inte-
grated them into the Cognitive Office
5
(two of them
5
See the Cognitive Office and the implemented Cogni-
tive Objects at www.lmt.ei.tum.de/team/kranz/videos.php
ROS Middleware
Visualization
Simulation
Intelligent
Environments
Proactive Services
HCI
Devices and
Smart Objects
Context Awareness
Wireless Sensor Networks
Location Awareness
Positioning and Navigation
Robotics
Simulation and Control
Mobile Interaction
Ubiquitous Control
Figure 2: The robotics middleware ROS in combination
with visualization and simulation tools can serve as a ba-
sis for various applications from different research fields.
implemented, one simulated).
Cognitive Cup. The Cognitive Cup is a coffee mug
augmented by self-awareness (see Fig. 3). It has an
accelerometer to detect its orientation and senses fluid
level and temperature of its content. The cup helps
robots to track its location and orientation by infrared
LEDs (invisibly for human eyes) built into the seam.
It communicates wirelessly with its environment via
RFID (for identification) and ZigBee (for real-time
sensor value transmission). Disclosing this informa-
tion can assist a grasping robot, e.g. to indicate that
the cup has to be handled with care in order not to
be spilled. More sophisticated services become pos-
sible, such as a robot automatically bringing a new
cup when the coffee is cold. The cup still supports
human affordances, as it resembles a normal cup in
shape, size and weight, and can even be put into the
dishwasher (after removing the socket).
Figure 3: The Cognitive Cup supports human affordances
(left) and robot-object interaction through infrared LEDs
(middle). Orientation, fluid level and temperature are trans-
mitted wirelessly, here in 3D visualization (right). See a
video at www.youtube.com/watch?v=mP3ZbPM9TVU.
Load Table. Information about non-augmented ob-
jects and related interaction can be gained through
adding cognition to furniture. Load sensing in a
wooden table provides three information primitives
about objects placed on it: weight, position, and type
of interaction based on the load signal shape over
time (Schmidt et al., 2002). The table becomes a
Cognitive Object disclosing information about other
objects, while the augmentation is entirely invisible
for human eyes. Multiple sensors are networked,
middleware-controlled and allow triangulation of ob-
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
166
ject positions based on measured load values. Addi-
tionally placed objects can be detected by comparing
the capacitive values to the previous state. The distri-
bution of cognition can assist robots in grasping non-
augmented objects which would otherwise be invis-
ible like e.g. glass. The table even detects whether
a glass is full or empty by monitoring the weight
change. A similar approach with capacitive sensors
has been shown in (Wimmer et al., 2007). Currently,
the table is integrated into the simulation only.
Whiteboard Cleaning Robot. Whiteboards are of-
ten used for quick notes. This robotic whiteboard
cleaner autonomously erases the whiteboard, choos-
ing the optimal path to only traverse areas containing
text. At the end, it moves automatically to its charg-
ing station. Before the cleaning procedure, the white-
board’s content is captured and saved as digital image
in order to preserve potentially important information.
The whiteboard is thus enhanced to an interactive sur-
face that autonomously makes information written on
it persistent. The text can be made digitally accessi-
ble and searchable using OCR. Special symbols can
provide additional functionality, e.g. the entire white-
board content can be emailed to a specified address
by drawing an envelope symbol. While the cleaning
robot in Fig. 4 is an early prototype, in the next iter-
ation the size will be reduced to fit into an everyday,
human-usable whiteboard eraser.
Figure 4: Left: The cleaning robot wiping a whiteboard.
Right: The robot navigates autonomously to find the most
efficient route and can be camera-tracked. See a video at
www.youtube.com/watch?v=hZxd-d1dDE4.
3.2 Context and Location Awareness
The localization of objects supports robot-object in-
teraction (e.g. an object being searched can announce
its location) and enable context-sensitive services
(e.g. sending a document to the nearest printer). The
Cognitive Office supports both active and passive lo-
calization. Context and location detection using sen-
sors can easily be integrated in the ROS middle-
ware. Passive presence detection (Kranz et al., 2006)
can e.g. be realized using PIR sensors or by infer-
ence from context information such as open doors or
windows with help of a HMM. We use this for au-
tomated HVAC, e.g. switching on the fan not only
temperature-based, but preferably when the worker
leaves the room in order not to distract him by the
noise. Another application would be to disable heat-
ing while the window is open to save energy.
We implement active location detection by
WLAN fingerprinting as indoor positioning system,
using triangulation and similarity search from a
database of location-annotated WLAN access point
signal strengths. Such a system can be the basis
for location-aware objects, up to indoor navigation
with help of a mobile phone. The Cognitive Office
and our simulation environment support all stages of
such a system’s development process. In the plan-
ning phase, the 3D simulation of the environment can
visualize the relative signal strength (RSS) of the ex-
isting WLAN base stations including reflections etc.,
e.g. using a color-coded map. This can help finding
positions for measuring fingerprints and adding them
to a database or even to find an optimal initial position
of WLAN access points. In a second phase, the posi-
tioning algorithm can be tested in the simulation and
provide valuable insights about its accuracy and relia-
bility before any real deployment, as not only the floor
plan and walls, but also all furniture and objects of the
real environment are accurately simulated. Changes
can be made easily and tests be rerun, which consti-
tutes an enormous advantage compared to real-world
tests in terms of costs and time.
3.3 Mobile Interaction
Mobile phone interaction approaches with the real
world have been presented, each with individual
drawbacks. Some require the augmentation of objects
with tags (visual or RFID), or special hardware, like
pointing using a laser (Rukzio et al., 2006). Magic-
Phone (Wu et al., 2010) uses the phone’s gyroscope
and accelerometer to detect in which direction it is
pointing has been presented. The phone can then be
used for remote control or interaction with the tar-
geted device, e.g. send slides to a projector or change
TV channels. The position of individual objects in
the room needs however to be known, as the phone’s
orientation is the only clue for detecting which device
it is pointing at. For the development and testing of
such a system, the whole environment would have to
set up, requiring physical space, time and costs.
Our approach using ROS and simulation tools
could perfectly support such a setup in simulation and
testing. The MagicPhone scenario has been realized
in simulation in our system to highlight the potential
of the proposed toolchain. The model of the mobile
ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -
Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through
Cognitive Objects
167
phone can be imported in simulation tool and inte-
grated into the virtual environment, which is an exact
model of the real-world rooms in which the system is
to be tested. Coming from the robotics area, ROS con-
tains methods for remotely controlling robots in an
environment, showing the robot’s exact field of view
in the simulation. These tools can similarly be used
for the simulation of a mobile phone carried by a per-
son. Pointing interactions are simulated by control-
ling the phone model using the PC keyboard, a multi-
axis controller or a MS Kinect camera. The object(s)
targeted by the phone can easily be detected by view-
ing the phone’s simulated camera image in the simu-
lation.
3.4 Extending Intelligent Environments
Intelligent environments or ‘smart spaces’ can be de-
fined as multi-user, multi-device, dynamic interaction
environments that enhance a physical space by virtual
services (Johanson et al., 2002). The aspect of be-
ing ‘dynamic’ comprises the smart space’s extension
over the borders of an augmented room to wherever
the user is, e.g. on the way from and to the office.
The Cognitive Office extends its functionality out
of the physical office space by receiving a notification
from the user’s smartphone when he approaches the
campus (detected via GPS or the WLAN SSID). The
office PC is thereupon booted via Wake-on-LAN, im-
portant appointments are shown on the calendar and
the email application is launched, ready to use when
the user reaches the office. The Cognitive Office also
takes care of a pleasant trip home: A traffic service
knowing the way home proactively searches for traf-
fic news on that route. In case of a traffic jam or ac-
cident, this news is reported even before the user is
getting in his car. If historically traffic jams occurred
frequently at a certain time (e.g. the rush hours), the
system proactively suggests the right time to leave the
office by self-learning.
3.5 Simulating Robots and Humans
Coming from the robotics domain, the ROS middle-
ware is perfectly suitable for monitoring, controlling
and evaluating interaction between robots and hu-
mans, objects and the environment. ROS incorporates
various methods for robotic control and interaction,
as well as drivers for components like vision systems.
We use ROS to control the PR2 in the Cognitive Office
(as seen in Fig. 5 in the 3D simulation).
Figure 5: Visualization of a PR2 robot’s hand camera views.
Due to the support of complex motion sequences
and the integrated physics engine, a bio-mechanical
human model can be integrated for simulating inter-
action with (intelligent) artifacts, robots or any other
objects in the virtual environment. ROS supports
complete control of arms, head and other parts of
the (robot) body, which can be adapted for a human
model. Different types of interaction, e.g. pressing
buttons, grasping objects, opening doors etc. can be
modeled and tested. Models can be controlled by
keyboard or another controller, e.g. a joystick. It is
imaginable to steer a human model through an ac-
curate model of a 3D simulation and use it to eval-
uate location-aware systems, e.g. presence detection
for proactive context-based services.
The visualization tool allows to monitor the
robot’s camera views as it moves in real time and thus
‘see the world through the robot’s eyes’. It is also pos-
sible to monitor the sensory input such as the robot’s
laser scanner, both from the real world and the simula-
tion. This allows to simulate a robot exploring an un-
known environment and to record the data it captures
before any deployment in the real world, which con-
stitutes valuable information for evaluating whether
e.g a service robot would get along in a domestic en-
vironment.
3.6 Interconnection
All examples just described are not individual appli-
cations put together in one room, but share the same
middleware and can communicate with each other.
The ROS publish/subscribe architecture lets arbitrary
components use any data and service available to the
middleware, allowing for a new stage of smart ser-
vices, and interaction between objects, robots and hu-
mans.
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
168
4 DESIGN GUIDELINES
We have presented our research goals to leverage
human-robot and robot-object interaction in the Cog-
nitive Office with help of Cognitive Objects. We
have presented examples from various contexts that
we have successfully developed, designed and tested
with help of this environment. In the remainder of this
paper, we draw first conclusions from our experiences
and deduce guidelines for the design of intelligent ar-
tifacts and environments that support interaction be-
tween humans, objects and robots. We begin with
comparing the abilities and characteristics of human
and machines, and afterwards give recommendations
for how objects and environments could be designed
to play out the strengths and compensate the weak-
nesses of humans and computerized systems for more
effective interaction, collaboration and cooperation.
4.1 Human-machine Comparison
As a basis for the later recommendations, the differ-
ences between humans and machines in terms of their
unequal abilities are described.
4.1.1 Human Abilities and Machine Abilities
The following classifications (Chapanis et al., 1951)
have been made in the context of aviation in order
to choose whether functions can be controlled by ma-
chines or are better operated by humans. These obser-
vations also apply for today’s computerized systems
and support function allocation in intelligent environ-
ments. Humans surpass machines in their ability to
detect small amount of visual and acoustic energy
perceive patterns of light or sound
improvise and use flexible procedures
store very large amounts of information for long
periods and to recall relevant facts at the appropri-
ate time
reason inductively
exercise judgment.
Machines appear to surpass humans with respect to
the ability to
respond quickly to control signals, and to apply
great force smoothly and precisely
perform repetitive, routine tasks
to store information briefly and then to erase it
completely
to reason deductively, including computational
ability
handle complex operations, i.e. to do many differ-
ent things at once.
4.1.2 People versus Machine-centered Views
The above abilities appear differently depending on
the point of view. From a machine-centered view,
people are inexact, often act illogically or emotion-
ally. Machines, by contrast, are always precise, or-
derly and logical. However, from a people-centered
view, machines lack essential abilities, such as cre-
ativity, imagination or adaptability, are ‘dumb’, while
humans are resourceful, creative and attentive to
changes. The fact that the strengths of one party are
weaknesses of the other (see Fig. 6) needs to be con-
sidered in the design of objects and environments con-
jointly used by humans and machines.
People are
Machines are
Machine-
centered
view
vague
disorganized
distractible
emotional
illogical
precise
orderly
undistractible
unemotional
logical
People-
centered
view
creative
compliant
attentive to changes
resourceful
able to make flexible
decisions based on context
dumb
rigid
insensitive to change
unimaginative
constrained to make consistent
decisions
Figure 6: Humans and machines have different abilities
which are depending on a people-centered or machine-
centered view strengths or weaknesses. Intelligent en-
vironments should both humans and computers support in
their strengths, and compensate their weaknesses. Tabular
representation according to (Norman, 1993).
4.2 Recommendations
Based on human and machine abilities and our experi-
ence in the Cognitive Office, we present initial recom-
mendations for the design of intelligent environments
and Smart Objects. We identify object and environ-
ment properties, and classify them according to their
importance for humans and robotic systems, as sum-
marized in Fig. 7. Objects and environments intended
for a conjoint use by humans and robots should re-
spect the recommendations from both categories.
4.2.1 Important Object Properties
Smart Objects should respect the following properties
to support interaction with robotic systems:
Visibility. The first precondition for interaction is
the object’s visibility to a robot, being a challenging
task for computer vision systems when the object is
transparent or has glossy and shiny surfaces. Small
objects might be hardly distinguishable. Low-light
ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -
Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through
Cognitive Objects
169
Environment Properties
Important
for robots
Location Awareness
Context/
Situation Awareness
Accessibility
Important
for humans
Comfort
Optionality of Technical Systems
Well-Being and Style
Affordance
Figure 7: Objects and environments for shared robot and
human use should support specific aspects, some of them
being especially important for humans or for computerized
systems.
environments, shadows or occlusions (e.g. because of
other objects in the line of sight or the robotic arm in
the way) can aggravate the problem. Objects can sup-
port such situations by ‘I am here’ self-disclosement,
e.g. by built-in RFID tags or infrared LEDs invisible
for humans.
Identifiability. Beyond being visible, objects need
to be identifiable by robots as specific object out of
many similar ones. This could likewise be realized
using RFID tags, as demonstrated in our Cognitive
Cup. Invisible technology like RFID helps maintain-
ing the object’s initial appearance, while visual tags
like QR codes can also be useful for humans, serving
as an indicator for embedded functionality.
Physical Interactability. Objects need to support
physical interaction with robots, e.g. be graspable by
robotic hands. Not only the shape of the object and
possible handles need to be formed accordingly, the
robot needs to know how to grasp it (e.g. a cup filled
with hot coffee needs to be grasped differently than an
empty cup, which can also be picked up on the handle
with the open side hanging down). Cognitive Objects
can assist interaction by disclosing information about
their internal state, e.g. whether a cup is empty or not.
Moreover, infrared LEDs can, invisibly for humans,
support pose estimation for a robot, which eases the
task of targeting it. The object can be adapted to the
needs of robot interaction, as long as it does not neg-
atively affect human interaction. The surface of our
Cognitive Cup is e.g. rougher than a porcelain mug,
in order to facilitate grasping by a robot hand.
Cooperation. Objects should cooperate in interac-
tion with robotic systems, instead of leaving cognition
and computation all to the robot. The distribution task
execution conforming the Ubiquitous Computing vi-
sion (Weiser, 1991) is more efficient in terms of com-
putational power and more effective. Cognitive Ob-
jects follow this paradigm (see (Möller et al., 2011)
for more detailed information).
Affordance. The term of affordance is placed at the
intersection of humans and machines in Fig. 7, as it is
equally important for both of them. In usability and
UI design, affordance denotes the ability of an object
to explain itself (Norman, 1990), i.e. to make intelli-
gible to an interacting person what it is for and how
to use it. Humans know based on their experience
– how to use e.g. a teapot and where to grasp it when
hot. Robot-usable everyday objects need special af-
fordances, which have been subject to research e.g. in
(Duchon et al., 1994) and (Fitzpatrick et al., 2003).
Affordances for robots can be realized by disclosing
their purpose and usage instructions in a machine-
understandable way. Cognitive Objects incorporating
sensors, actuators, computation and communication
are a means to provide this semantical knowledge.
The augmentation for robot usability must not go
along with replacement, but coexistence of human
affordances. Just as human affordances are ‘invisible’
for robots, parts relevant only for technical systems
(like e.g. antennas) should be hidden from human
eyes.
For human usage, Smart Objects should respect
Usability. To be practical and simple to use for hu-
mans, interaction design, cognitive psychology and
cultural factors should be considered in the design
progress. Consider an ‘intelligent’ cup signaling the
temperature of its content. While the cup internally
possibly works with a numeric temperature scale, the
temperature might be glimpsed more intuitively by a
person if it is translated into a color scale (red = hot,
blue = cold), instead of just showing the numeric tem-
perature value on a display.
Convenience. Augmented objects must not re-
duce convenience compared to conventional, non-
augmented objects. When e.g. the battery is drained,
the object should still offer its non-technical function-
ality, instead of not being usable at all.
4.2.2 Important Properties for Environments
For an intelligent environment being used by robotic
systems, the following factors are of importance:
Location Awareness. For a human, it is self-
evident to know whether he is in the kitchen or in the
living room, while for a robot, it isn’t. Intelligent en-
vironments suitable for robots thus should support lo-
cation awareness for computerized entities. They can
e.g. be equipped with multiple WLAN base stations
to enable WLAN fingerprinting, or offer positioning
using DECT (Kranz et al., 2010a), especially for Cog-
nitive Objects.
Context Awareness. Intelligent environments aim
at facilitating people’s everyday life by context-aware
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
170
services and applications. The environment can sup-
port machines in providing this context awareness,
e.g. by providing the respective sensors to detect dif-
ferent sorts of events, such as presence by motion sen-
sors, opening or closing of windows, doors and draw-
ers by contact switches, etc.
Accessibility. In a mixed human-robot environment
all rooms and areas must be accessible by robotic
locomotion systems. While some robots are able to
use stairs, for most wheel-based robots, stairways
are an insuperable obstacle. The width of doorways,
the space between furniture or missing space to turn
around can likewise limit robots to access a desired
place. The height of desks, cupboards or switches
needs to match the robot’s ability to move and stretch
in order to enable interaction. As vision-based
recognition systems perform worse in recognizing
objects in bad lightning conditions, the installation of
additional light sources could produce relief.
Important for humans are the following factors in the
design of an intelligent environment:
Comfort. Living spaces and work places should as-
sist human users by offering a practical environment,
i.e. all needed objects in a certain situation are placed
within reach, the furnishings and facilities fit the spe-
cific needs, etc. For additional comfort, context-
aware services and proactive functionality simplify
repeating tasks and remove complexity. Technical
systems should take over activities according to their
strengths (see Sec. 4.1.1), such as routine tasks, sim-
ple control actions like switching on the light on ac-
tivity, but leave complex or creative tasks to humans.
Optionality of Technical Systems. Technical sys-
tems should not entirely replace conventional ways of
task completion, following the principle of ‘augmen-
tation instead of automation’. Proactive behavior of
an intelligent home should not manifest in patroniz-
ing inhabitants, but rather in offers and suggestions
they can adopt or silently ignore. The ability to over-
ride the system’s ‘intelligence’ needs to be given at
every time.
Well-being and Style. People want to feel com-
fortable in their environment, recognizable by the
amounts of money spent for attractiveness and de-
sign. Besides functionality, the emotional relationship
to objects and the so-called ‘visceral design’ (Nor-
man, 2003) become increasingly important. A chair is
no longer an object for comfortable sitting but some-
times a piece of art. A similar trend is observable in
electronics, with regard to thoroughly and artistically
designed phones, computers or home entertainment
components, using expensive materials like glass and
aluminum. Environments designed for machines and
humans cannot neglect this requirement any more.
Although being suitable for robot use, such environ-
ments need to satisfy the human wishes of a pleasur-
able, nicely designed home.
5 CONCLUSIONS
We have presented an approach for supporting robotic
systems in interaction with physical objects by Cog-
nitive Objects, with which we have opened a com-
pletely new research field at the intersection of per-
vasive computing and personal robotics. Cognitive
Objects are real-world artifacts embodied in an in-
teraction that include sensors, actuators, communica-
tion, and computation, cooperating with humans and
robots to support task execution. Today’s robotic vi-
sion systems, and overall interaction capabilities, are
not sufficient for arbitrary object recognition, hin-
dering a full integration of personal robotics in real-
world environments. By cooperatively disclosing in-
formation about themselves, Cognitive Objects facil-
itate the recognition of and interaction with any kind
of object by robots and thus are a step towards more
comprehensive robot-object interaction. At the same
time, Cognitive Objects are not singularly designed
for machine-machine interaction, but remain equally
usable by humans, maintaining usability and affor-
dances.
We demonstrated the potential for shared object
usage by humans and robots in our Cognitive Office,
an intelligent environment using a robotic middle-
ware, and showed the potential for supporting mobile
interaction, context and location awareness.
We have given initial recommendations for the de-
sign of objects and environments with regard to a
shared usage by robots and humans. Cognitive Ob-
jects can sustainably support and stimulate research
in personal and robot interaction, as well as ubiqui-
tous computing, as they provide valuable ground truth
data to robots and facilitate interaction for both hu-
mans and robots. In the future, we plan to integrate
further Cognitive Objects into our intelligent environ-
ment, focusing on supporting ambient assisted living
and personal robotics scenarios and their evaluation
in complex interactions.
Resources. Source code, documentation and data
sets of the presented systems and environment are
shared via https://vmi.lmt.ei.tum.de/ros/.
ROBOTS, OBJECTS, HUMANS: TOWARDS SEAMLESS INTERACTION IN INTELLIGENT ENVIRONMENTS -
Supporting Complex Cooperative Interactions between Humans and Technical Systems in Real World Scenarios through
Cognitive Objects
171
ACKNOWLEDGEMENTS
This work has been funded in parts from the German
DFG funded Cluster of Excellence ‘CoTeSys Cog-
nition for Technical Systems’ and the German BMBF
funded project ‘GEWOS – Gesund wohnen mit Stil’.
REFERENCES
Beigl, M. and Gellersen, H. (2003). Smart-its: An embed-
ded platform for smart objects. In In Proc. Smart Ob-
jects Conference (SOC 2003, pages 15–17.
Chapanis, A., Frick, F. C., Garner, W. R., Gebhard, J. W.,
Grether, W. F., Henneman, R. H., Kappauf, W. E.,
Newman, E. B., and Williams, A. C. J. (1951). Hu-
man engineering for an effective air-navigation and
traffic control system. National Academy of Sciences
Archives.
Duchon, A. P., Warren, W. H., and Kaelbling, L. P. (1994).
Ecological robotics. Adaptive Behavior, 6.
Fitzpatrick, P., Metta, G., Natale, L., Rao, S., Sandini, G.,
and S, G. (2003). Learning about objects through ac-
tion - initial steps towards artificial cognition. In In
Proc. ICRA, pages 3140–3145.
Holmquist, L., Gellersen, H., Kortuem, G., Antifakos, S.,
Michahelles, F., Schiele, B., Beigl, M., and Maze, R.
(2004). Building intelligent environments with Smart-
Its. Computer Graphics and Applications, IEEE,
24(1):56–64.
Johanson, B., Fox, A., and Winograd, T. (2002). The inter-
active workspaces project: experiences with ubiqui-
tous computing rooms. Pervasive Computing, IEEE,
1(2):67–74.
Kranz, M., Fischer, C., and Schmidt, A. (2010a). A com-
parative study of DECT and WLAN signals for indoor
localization. In PerCom 2010, pages 235 –243. IEEE.
Kranz, M., Holleis, P., and Schmidt, A. (2006). Ubiquitous
Presence Systems. In SAC ’06: Proceedings of the
2006 ACM symposium on Applied computing, pages
1902–1909. ACM.
Kranz, M., Holleis, P., and Schmidt, A. (2010b). Embedded
Interaction: Interacting with the Internet of Things.
IEEE Internet Computing, 14:46–53.
Kranz, M., Linner, A., Ellmann, B., Bittner, A., Bock, T.,
and Roalter, L. (2010c). Robotic Service Cores for
Ambient Assisted Living. Pervasive Health 2010.
Kranz, M. and Schmidt, A. (2005). Prototyping Smart
Objects for Ubiquitous Computing. In Workshop on
Smart Object Systems, UbiComp.
Kranz, M., Schmidt, A., Rusu, R., Maldonado, A., Beetz,
M., Hornler, B., and Rigoll, G. (2007). Sensing
Technologies and the Player-Middleware for Context-
Awareness in Kitchen Environments. In Proc. of INSS,
pages 179 –186.
Kranz, M., Schmidt, D., Holleis, P., and Schmidt, A. (2005).
A Display Cube as a Tangible User Interface. Adj. In
Proc. of UbiComp 2005. Springer.
Laerhoven, K. V., Schmidt, A., and Gellersen, H.-W.
(2002). Pin&play: Networking objects through pins.
In UbiComp ’02, pages 219–228. Springer-Verlag.
Linner, T., Kranz, M., Roalter, L., and Bock, T. (2010).
Compacted and industrially customizable ambient in-
telligent service units: Typology, examples and per-
formance. In 2010 Sixth International Conference on
Intelligent Environments, pages 295–300.
Möller, A., Roalter, L., and Kranz, M. (2011). Cog-
nitive Objects for Human-Computer Interaction and
Human-Robot Interaction. In 6th ACM/IEEE Interna-
tional Conference on Human-Robot Interaction.
Norman, D. A. (1990). The design of everyday things. Dou-
bleday.
Norman, D. A. (1993). Things that make us smart: de-
fending human attributes in the age of the machine.
Addison-Wesley Pub. Co.
Norman, D. A. (2003). Emotional Design: Why We Love
(or Hate) Everyday Things. Basic Books.
Roalter, L., Kranz, M., and Möller, A. (2010). A middle-
ware for intelligent environments and the internet of
things. In Ubiquitous Intelligence and Computing,
volume 6406 of Lecture Notes in Computer Science,
pages 267–281. Springer Berlin / Heidelberg.
ROS (2010). An open-source, meta-operating system
for your robot, robot operating system middleware.
http://www.ros.org.
Rukzio, E., Leichtenstern, K., Callaghan, V., Holleis, P.,
Schmidt, A., and Chin, J. (2006). An experimental
comparison of physical mobile interaction techniques:
Touching, pointing and scanning. In Dourish, P. and
Friday, A., editors, UbiComp 2006: Ubiquitous Com-
puting, volume 4206 of Lecture Notes in Computer
Science, pages 87–104. Springer Berlin / Heidelberg.
Schmidt, A., Strohbach, M., van Laerhoven, K., Friday,
A., and Gellersen, H.-W. (2002). Context acquisition
based on load sensing. In Borriello, G. and Holmquist,
L., editors, UbiComp 2002: Ubiquitous Computing,
volume 2498 of Lecture Notes in Computer Science,
pages 161–192. Springer Berlin / Heidelberg.
Siio, I., Rowan, J., and Mynatt, E. (2003). Finding objects
in "strata drawer". In CHI ’03: CHI ’03 extended ab-
stracts on Human factors in computing systems, pages
982–983. ACM.
Ullmer, B. and Ishii, H. (2000). Emerging Frameworks for
Tangible User Interfaces. IBM Syst. J., 39(3-4):915–
931.
Weiser, M. (1991). The Computer For The 21st Century.
Scientific American, 265(3):66–75.
Williams, B. C. and Nayak, P. P. (1996). Immobile Robots
AI in the New Millennium. AI Magazine, 17(3):16–
35.
Wimmer, R., Kranz, M., Boring, S., and Schmidt, A.
(2007). Captable and capshelf - unobtrusive activity
recognition using networked capacitive sensors. In
Proc. Fourth International Conference on Networked
Sensing Systems INSS ’07, pages 85–88.
Wu, J., Pan, G., Zhang, D., Li, S., and Wu, Z. (2010). Mag-
icphone: pointing & interacting. In Ubicomp ’10:
Proceedings of the 12th ACM international confer-
ence adjunct papers on Ubiquitous computing, pages
451–452. ACM.
PECCS 2011 - International Conference on Pervasive and Embedded Computing and Communication Systems
172