A SCIENCE OF INTERACTION
A Multidimensional Canvas
Robert Spence
Imperial College London, U.K.
Keywords: Interaction, information visualization, visual analytics, browsing, design actions.
Abstract: A recently proposed search for a science of interaction will involve the creation of a multidimensional
canvas composed of many features: complex perceptual and cognitive processes, the many approaches to
interaction design, the many models of interaction and the vast range of interaction modalities. In any
attempt to meet this challenge this paper urges emphasis on precise definitions, especially for visualization
and interaction, and reports the results of an exploratory study.
1 INTRODUCTION
In a publication (Thomas & Cook, 2005) setting out
an agenda for research and development, the
Department of Homeland Security in the United
States encourages the creation of a new science of
interaction to support Visual Analytics, the latter
discipline being defined as:
Visual Analytics: The science of
analytical reasoning facilitated by
interactive visual interfaces.
Because a major feature of this paper is the stress
that it lays on the need for precision in concept
definitions it must be pointed out that, unlike many
definitions which have incorrectly separated the
letters v-i-s-u-a-l from the term ‘visualization’, the
aforementioned report on Visual Analytics does
acknowledge forms of encoding (e.g., aural, tactile)
other than visual.
In some respects the disciplines of Visual
Analytics and Information Visualization are hard to
disambiguate one from the other. However, because
the definition of Information Visualization is
relevant to the following discussion we repeat a
common dictionary definition here:
Visualization: to form a mental model
of something.
Though apparently controversial, this definition
removes any immediate consideration of technology
and implies that a ‘visualization’ is not something
that appears on a computer display. Indeed, in its
mention of a mental model it supports the concern of
visual analytics with cognitive science.
A Challenging Goal. In proposing a search for a
science of interaction, the report on Visual Analytics
describes an exceedingly challenging goal, assuming
that ‘science’ refers to a body of knowledge, and
hopefully one whose formulation is supportive of
organised thought and design. Yi et al (2007) have
recently addressed that challenge. What follows in
this paper are some considerations regarding the
challenge and an example illustrating one attempt to
address the complexity involved
Precision. The prospect of making progress towards
an entirely new science is fraught with difficulty in
the absence of clear definitions of the concepts
involved. Even if the sought-after science does not
materialize as coherently as one would wish, clear
definitions are in any case essential to support
organised thought about a subject and vital for its
advancement.
2 INFORMATION
VISUALIZATION
Based on the above definition of visualization it is
helpful to identify key features of a system designed
to support information visualization. Those features
are (Spence, 2007):
415
Spence R. (2008).
A SCIENCE OF INTERACTION - A Multidimensional Canvas.
In Proceedings of the Third International Conference on Computer Graphics Theory and Applications, pages 415-418
DOI: 10.5220/0001099604150418
Copyright
c
 SciTePress

1. Representation, in which raw or derived
data (Tweedie, 1997) is encoded by
graphical, aural, tactile or olfactory means.
2. Presentation, which involves the selection,
and the spatial and temporal arrangement,
of represented data for display, where
‘display’ is understood to include all means
of presenting data.
3. Interaction, leading to changes in
representation and presentation that have
the potential to support the generation of
insight into data.
While these three features can be separated
conceptually, the potential for interplay between
them is of course rich and available for beneficial
exploitation by the interaction designer. Two
comments should be made. First, as argued by
Furnas (2006), the distinction between
representation and presentation is important: the two
should not be merged. It should also be pointed out
that interaction as usually understood is not an
essential component of a system designed to support
visualization, otherwise all pre-computer
representations (e.g., of Minard, Snow and
Nightingale) and all static representations, whether
printed or electronic, would be excluded from
consideration.
3 INTERACTION
In the search for an interaction science it is essential
to establish a classification of interaction types.
Three such classes are
Continuous Interaction. In continuous interaction
the value of some variable is changed continuously
and the display of data is changed accordingly and
usually continuously. An example is provided by a
display in which the value of some parameter of a
model is continuously varied – manually or
autonomously – and the data corresponding to each
parameter value is displayed, again normally
continuously.
Stepped Interaction. In stepped interaction a single
action such as a mouse-click causes a discrete
change in the presented representation, either
involving essentially the same data or a move to a
completely new location in discrete information
space. A very familiar example is provided by the
transition from one web page to another following
interaction with a displayed menu item.
Passive Interaction. Except for the rare occasions
in which continuous interaction is employed, and the
few milliseconds in which an action such as a mouse
click is executed, most of a user’s time is spent in
passive interaction, simply examining and
interpreting representations of data presented
visually, aurally or tactilely. With the graphical
encoding of data, for example, a great deal of eye
movement is involved. An example is provided by a
person undertaking a visual examination of Minard’s
map; another involves an online purchaser carefully
studying the options available before proceeding to
make a selection. There should be no problem with
the term ‘passive interaction’: the prefix ‘inter’
means ‘between’ or ‘among’, and ‘act’ is defined as
the process of doing something. We speak, for
example, of social interaction, an exceedingly
complex process in which there is no need for any
physical act to take place; similarly it is appropriate
to speak of passive interaction to refer to the
complex visual and cognitive actions on the part of
the person studying Minard’s map or a temporarily
static display.
Two comments may help to remove any doubt as
to the validity of the class of passive interaction.
First, it is important to note that passive interaction
does not imply a static display such as a
conventional map: a visual display can usefully be
dynamic and designed in such a way that a user can
derive considerable insight simply by watching it
(Colgan et al, 1995; Wittenburg et al, 2003).
Second, there is a common misunderstanding
that a static display is somehow inferior to an
interactive one, a view that might unfortunately be
reinforced by an emphasis on ‘interaction’ as it is
conventionally interpreted. One only has to
compare use of the Dynamic Queries interface
(Williamson & Shneiderman, 1992) with a static
instance of the Attribute Explorer (Spence &
Tweedie, 1998) to realise that, in the former, a great
deal of interaction is required to build up a mental
model of the underlying data, whereas the latter
provides considerable insight without any need to
change the view.
4 HUMAN INTENT
The need to recognise the involvement of the human
user of a system designed to support visualization
has already been acknowledged by the very
definition of visualization. However, to work
towards some useful classification there is a need to
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
416

examine the perceptual and cognitive aspects of that
user in more detail. In the context of a search for a
science of interaction a first step in this direction has
been made by Yi et al (2007) who draw attention to
the importance of human intent.
This human characteristic can be further
subdivided by considering the activity of browsing.
Definitions of browsing are notorious for their lack
of precision so we offer the following definition:
Browsing: the perception, interpretation and
evaluation of content,
a definition which, on its own, identifies the Gulf of
Evaluation in Norman’s Stages of Action (Norman,
1988) and often involves little or no conscious
cognitive effort (Potter, 1999). Unfortunately, in the
literature, the term ‘browsing’ carries with it the
connotation of casual intent. Instead, and broadly in
agreement with Foster and Ford (2003), de Bruijn &
Spence (2007) define three classes of browsing, each
simply described by the presence or absence of
intent and by the awareness or otherwise of a goal:
• Search browsing (SB) is intentional, and
the user is aware of a goal;
• Opportunistic browsing (OB) is
intentional, but the user is unaware of a
goal;
• Involuntary browsing (IB) is
unintentional and the user is unaware of a
goal.
An example of SB is demonstrated by the
familiar use of the Web to locate either specific or
initially ill-defined information. OB is characterised
by the attitude “I wonder what’s there? Let’s have a
look”. IB occurs all the time that a user is
conscious, normal rapid eye movements (typically 3
per second) providing the change of visual
stimulation of the eye (the ‘change in the world’ in
Norman’s Action Cycle) and which is continuously
categorised and consolidated (Potter, 1999). As an
example, all three forms of browsing are almost
certainly invoked when using the Web to search for
a present for Mother’s Day.
5 DIMENSIONS
Although our definition of browsing provides
classes of intent, that is only one of many
dimensions within which the optimum choice of an
interaction scheme for a given application needs to
be considered. In addition to human intent there is
the perceptual and cognitive dimension, a variety of
models of interaction (e.g., GOMS), the many
modes of interaction (e.g., eye-gaze, mouse, voice)
and a range of design levels. The latter can range,
for example, from the use of sketching to support
ideation and creativity (Craft, 2006; Buxton, 2007)
through the use of Patterns (Borchers, 2000) and
guidelines, to other complementary concepts such as
Design Actions (de Bruijn & Spence, 2007) that
provide detailed and often quantitative advice to the
interaction designer. The science of interaction
would seem likely, therefore, to occupy a
multidimensional canvas and one, moreover, which
is still largely unpainted. It seems inevitable that
many exploratory investigations will take place in
the search for such a science.
The result of one exploratory study, described in
the following section, is a potentially useful
framework associated with three of the many
relevant dimensions, in that it links the human
behaviour supported by an interface, the cognitive
theory underlying that behaviour and the design
decisions made by an interaction designer.
6 DESIGN ACTIONS
The framework is shown in Figure 1. Each of the
three components is identified both generically and
by the specific example to be used for illustration.
The human behaviour for which an interface must be
designed is at the centre, and in the illustrative
example is opportunistic browsing. The behaviour
invokes perceptual and cognitive processes, and in
our illustration the relevant example is that of human
visual processing as represented by the model of
Conceptual Short-term Memory (Potter, 1999). On
the right are Design Actions which inform the
interaction designer: their nature is described below.
The motivation for the proposed framework is
the perceived advantage that an interaction designer
need not be familiar with or understand the
underlying cognitive theory: that theory is ‘reflected’
via human behaviour into Design Actions which are
couched in the language of the interaction designer.
To illustrate the nature of Design Actions we
refer to an example chosen by de Bruijn and Spence
(2007): the opportunistic browsing, on a PDA, of
news items (de Bruijn & Tong, 2003). Each news
item is represented by an appropriate image and very
few words, and they are presented in sequence by
Rapid Serial Visual Presentation. The question
“how rapid?” is one that Design Actions (DAs) can
A SCIENCE OF INTERACTION - A Multidimensional Canvas
417

Cognitive Theory
Example: Conceptual
Short-term Memory
Human behaviour
Example: Opportunistic
Browsing
Design
Actions
Examples:
Rates of
image
presentation
Figure 1: The Framework involving Design Actions.
answer. For example, one DA is appropriate if a
satisficing strategy is adopted by the user, and
suggests a maximum presentation rate of 10 per
second. Another DA is relevant to an ‘optimising’
strategy, for which the rate could be as high as 2 per
second. Yet another DA is relevant if a user wants
to preview all news items and then choose one after
all have been considered: here the rate is about 1
item per second. Thus, the relevant DA depends
upon the interaction designer’s assessment of the
user’s intent. As well as providing detailed advice
each DA describes the ‘upsides’ and ‘downsides’ of
its application as well as associated issues.
7 CONCLUSIONS
Three conclusions with regard to the search for a
science of interaction can be drawn. First, there is a
need for such a science to be based on concept
definitions which are precise. Second, the
definitions of visualization and interaction in
particular need to be sufficiently inclusive as not to
exclude many of the common modes of interaction.
Third, that the search for a science of interaction will
require many exploratory studies over a considerable
period of time. Thus, the Design Actions concept
described above may well be only a tiny speck of
paint on the forbidding multidimensional canvas that
will become the science of interaction.
ACKNOWLEDGEMENTS
I thank the authors of Yi et al (2007) for providing a
preview of their paper. I’m especially grateful to Dr.
Oscar de Bruijn for many discussions about
interaction design.
REFERENCES
Borchers, J.O. (2000) A Pattern Approach to Interaction
Design, New York, John Wiley & Sons.
Buxton, B. (2007) Sketching User Experiences,
Amsterdam, Morgan Kaufmann.
Colgan, L., Spence, R. and Rankin, P.R. (1995) The
Cockpit Metaphor, Behaviour and Information
Technology, 14, 4, pp.251-263.
Craft, B. and Cairns, P. (2006) Using Sketching to Aid
the Collaborative Design of Information Visualisation
Software – A Case Study, HWID’06 Human Work
Interaction Design: DESIGNING FOR HUMAN
WORK, Madeira, Portugal, 13-15 February 2006.
de Bruijn, O. and Tong, C.H. (2003) M-RSVP: mobile
web browsing on a PDA, in O’Neill, E., Palanque, P.
and Johnson, P. (eds) People and Computers –
Designing for Society, London, Springer, pp.297-311.
de Bruijn, O. and Spence, R. (2007) A New Framework
for Theory-based Interaction Design, applied to
serendipitous information retrieval, ACM Transactions
on Computer-human Interaction; (in press).
Foster, A. and Ford, N. (2003) Serendipity and
Information Seeking: an empirical study, Journal of
Documentation, 59, pp.321-340.
Furnas, G.W. (2006) A fisheye follow-up: further
reflections on focus + context, Proceedings CHI2006,
pp.999-1008.
Norman, D.A. (1988) The Design of Everyday Things,
New York, Doubleday.
Potter, M. (1999) Understanding sentences and scenes: the
role of conceptual short-term memory, in Coltheart, V.
(ed) Fleeting Memories: Cognition of Brief Visual
Stimuli, Cambridge, MA, MIT Press, pp.13-46.
Spence, R. and Tweedie, L. (1998) The Attribute
Explorer: information synthesis via exploration,
Interacting with Computers, 11, pp. 137-146.
Spence, R. (2007) Information Visualization: design for
interaction, Prentice Hall.
Thomas, J.J. and Cook, K.A. (2005) Illuminating the Path,
IEEE.
Tweedie, L.A. (1997) Characterizing interactive
externalizations, ACM, Proceedings CHI’97, pp.375-
382.
Williamson, C. and Shneiderman, B. (1992) The dynamic
housefinder: evaluating dynamic queries in a real
estate information exploration system, ACM,
Proceedings SIGIR’92, pp. 339-346.
Wittenburg, K., Forlines, C., Lanning, T., Esenther, A.,
Harada, S. and Miyachi, T. (2003) Rapid serial visual
presentation techniques for consumer digital video
devices, ACM, Proceedings UIST, pp. 115-124.
Yi, J.S., Kang, Y.A., Stasko, J.T. and Jacko, J.A. (2007)
Toward a Deeper Understanding of the Role of
Interaction in Information Visualization, Proceedings
InfoVis 2007.
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
418