Exploring the Virtuality Continuum Frontiers: Μultisensory and

Magical Experiences in Interactive Art

Nefeli Georgakopoulou

, Dionysios Zamplaras

, Sofia Kourkoulakou

Chu-Yin Chen

and François Garnier

INREV Laboratory, Paris 8 University, Paris, France

EnsadLab Spatial Media, PSL Université, Paris, France

Keywords: Interaction Design, User Interface Design, Tangible Interfaces, Smart Material Interfaces, Haptics, Active

Sensing, Immersive Experience, Multisensory Experience, Mixed Reality, Magical Reality.

Abstract: This article is an effort to approach certain aspects of the evolution of user interface design, as well as the

design and aesthetics of interactive works of art. Interactive works attempt to connect the viewer with the

work and invite him to deploy his senses and his body. But many researchers and artists are concerned by

the fact that interface technologies tend to put us out of touch with ourselves, so they try to create

interactions that take place in our periphery, drawing our attention rather than demanding it. Taking as a

starting point the sense of touch, whose importance has been widely analysed in philosophy,

phenomenology and aesthetics, we describe the modalities of active touch and active sensing. We then

propose that with the use of physical materials and tangible interfaces in interactive artworks we can

achieve more magical experiences that engage us in a multi sensorial way. As an example we are describing

the concepts around the mixed reality installation VitRails.

1 INTRODUCTION

Through the use of different types of human

computer interfaces, interactive works seek to create

various multimodal interactions. The resulting multi

sensorial experiences challenge and defy our

perceptual channels. And while the visual channel

seems to predominate in our culture, the conception

and design of interfaces and connected objects has

long embraced the idea of alternative and more

natural ways of producing additional and

complementary feedback through the involvement of

our other perceptual channels.

This research is an effort to analyze and

understand the quality of different Human-

Computer-Interactions (HCI) aiming in more “calm”

technologies and “natural” interactions through the

shift from passive to active sensing. We suggest that

a change from the pictorial images of graphical user

interfaces towards the abstraction of touch and

materiality in tangible user interfaces can result in

more interesting immersive and magical experiences

in art.

In the following sections, we try first to state

some basic ideas around the characteristics of

interactive art and the way we perceive it. We then

suggest that the active use of our other senses can

engage us in multi-sensory immersive experiences.

We also propose the enrichment of these experiences

with physical materials, as this can create more

natural bonds with the physical world, thus adding

the layer of magic to the real virtual continuum. We

end with the description of the mixed reality

installation VitRails

2 INTERACTIVE ART & NEW

TYPES OF HCI

Art and cultural practices in general have always

been related to the tools and media provided.

Nowadays, new ways of using existing technology,

as well as new technological advances, come to offer

new tools and new media at the disposition of artists

and creators. With the arrival of the digital era and

https://collectifcontinuum.wixsite.com/continuum

Georgakopoulou, N., Zamplaras, D., Kourkoulakou, S., Chen, C. and Garnier, F.

Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art.

DOI: 10.5220/0007573901750182

In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2019), pages 175-182

ISBN: 978-989-758-354-4

175

the computers, the borders between different forms

of art merge. The image becomes more modulable

and easy to manipulate and transform, sometimes

even more abstract, challenging anew the notions of

time and space, putting the viewer / spectator in the

center of action and minimizing the distance

between subject and object. As the technological

tools in our disposal continue to evolve, the borders

between the different forms of expression also

become more blurry (Manovich, 2001).

Interactive works are a form of expression that

attempt to connect the viewer with the work, involve

him and make him participate in the events, to make

him interact with his body and his mind. The human

body and the way it operates, which in some forms

of artistic expression are less important, become

now fundamental notions in the evolution of the

narrative. Interactive works often reveal the body

and the senses during the process of the experience.

Moreover, interactive art conceives the work as

“open” to the various manipulations of the visitor,

and thus in a certain way it is proposed as

“unfinished” by its creator. To the action of the

spectator, the work becomes phenomenologically

different and same goes for the experience. This is

obviously not without asking questions about the

limits of the work or the role of interaction, or even

the role of the artist himself (Balpe, 2000).

The interaction with our environment, whether

real or virtual, becomes possible through the body,

through the use and coordination of expressions,

movements, or speech. The response of the

environment is received at the emotional and mental

levels via our five main senses.

From the human perception point of view, our

five primary senses constitute our sensory modalities

and modes of communication. When it comes to

virtual and interactive works, we use the word

modality to describe the input or output modes of

interaction. This means that an interaction modality

can be defined as a particular and concrete form of a

communication mode (Appert, 2016). From these

notions, we can deduce that an interactive work is

multimodal when it has several modalities as input

or output. The notion of modalities is of great

importance in the fields of interaction and interface

design.

Most interface technologies tend to separate the

functions of our physical body, bringing us near to a

state of disintegration by putting us out of touch with

ourselves and the environment around us (McLuhan,

1964). Moreover, the reception of the majority of the

information that comes from the digital world is

made through devices that we carry on with us, that

tend to remove us from our surroundings and

demand our attention. Nevertheless, many

researchers and artists are concerned by this fact and

try to create more “calm” technologies, where the

interactions take place in our periphery, drawing our

attention rather than demanding it.

We are now witnessing new opportunities in

interactive art especially with the introduction of

new materials and interface technologies such as

Tangible User Interfaces (Ishii, 2008), Organic User

Interfaces (Vertegaal and Poupyrev, 2008), and

Reality-Based Interfaces (Jacob et al., 2008) as well

as with the new vision called Smart Material

Interfaces (Minuto and Nijholt, 2013) - TUI, OUI,

RBI and SMI respectively.

All of these interaction styles try to take

advantage on users' pre-existing knowledge of

everyday life, except of SMIs which are trying to

surpass classical patterns of interaction and abandon

the “digital feeling" for a more analog and

continuous type of interaction through methods that

focus on the properties of materials (Minuto and

Nijholt, 2013). This can add to interactive art and

design a new layer of magic that transcends what we

are already used to.

Here we are not going to discuss about magic as

the possess of supernatural forces, but rather as the

surprising phenomena that are hidden/existing in

nature

, and with the help of science and technology

we can reveal them. We are going to focus on this

magical and exciting quality of the combination of

techno science with art which can make something

seem different from ordinary things. We propose

that with the use of physical materials in interactive

artworks we can achieve more fluid experiences that

engage us in a multi sensorial way. In the next

section, we will be discussing the way we actively

sense and perceive the interactive works, taking as a

starting point haptics and the sense of touch.

3 FROM HAPTICS TO ACTIVE

SENSING

The word haptics comes from the greek word

haptein/háptô, meaning touch

. In the field of art and

aesthetics, this word has been used by the autrichien

art historian AloïsRiegl who, in his work “Late

https://www.larousse.fr/dictionnaires/francais/magie/48

531

Le Grand Robert. 2008, Dictionnaires Le

Robert/Sejer.500/117/200808/1LGR264CF-CD

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

176

Roman Art Industry” made the distinction between

the optical and the tactile modes of representation

and suggested an evolution from one pole to the

other (Zerner, 1976). To these days, the word haptics

is often used as the term to describe the kinesthetic

and cutaneous sensory feedback.

Historically, the sense of touch has often been

considered essential to our perception of the

environment and the objects around us. According to

Aristotle, who affirmed that things are tangible prior

to being visible or audible, the touching, since it

affects the whole body, can be considered as the

sense of embodiment “par excellence” (Massie,

2013), to the extent that it places the man closer to

the world and the things closer to him (Aristote,

1993).

According to Merleau-Ponty, but also many

other theorists, like Varela, it is recognized that

action and perception should be considered together

when analyzing behavior and cognition (Merleau-

Ponty, 1945; Varela et al, 1993).

Gibson, in his work, made the distinction

between being touched and the act of touching,

which he named active touching. These two notions

made a rather obvious, but very important

observation, that objects can be identified and

distinguished more precisely if a person is allowed

to move their hands and fingers.

This concept of exploration is relative to the

theories of Merleau-Ponty. Our capacity of

understanding and knowing our environment as

human beings lies more to a combination of active

processes. This should remind us that the interest of

an active touching exploration resides, not only in

the perceptual experience in itself, but also in the

understanding of the way we form the knowledge of

our surroundings and our world (Prescott et al,

2011).

Regarding the tactile perception, Loomis and

Lederman (1986) attempted an analysis of

interpreting information through this modality.

According to them, a person can use multiple modes

of exploration for each characteristic, when trying to

identify an object through touch. They state that

there are two fundamental and distinct senses that

together provide us with a sense of touch: the

cutaneous sense that provides an awareness of the

stimulation and the kinesthesis that provides an

awareness of the relative positioning of the body.

The form that combines these two senses is

described as the haptic perception, and it is the mode

of exploration and understanding of our environment

which interests this research.

In an attempt to generalize these ideas around

active touch to our other senses, many philosophers

consider that all the senses function in a similar way

to the touch and that they are based on the same

principles (Levine and Touboul, 2015). For french

philosopher and scientist Réné Descartes, for

example, to see is first and foremost to touch. As the

blind "sees" with his hands, a metaphor to state that

the object in this case is to the hand what the ray of

light is to the eye (Descartes, 1966).

Sensing and perceiving is in constant and co-

dependent relation with our body and our spirit. Our

meeting with the world happens through our bodies,

as this is our way of expressing ourselves and our

intentions in a visible form, a place where our spirit

takes a form, a way to understand our surroundings.

This incarnated consciousness is directly linked to

our body and the world, because our existence as

“bodies” cannot be separated from the existence of

our world (Merleau-Ponty, 1945). In that way, a

purely personal reality is created through our way of

perceiving and the world becomes unique for each

human being, outside of any objectivity.

4 MATERIALS AS INTERFACES

As discussed above, unfortunately, technologies

today only engage a small portion of our senses,

deeply neglecting the full range of material qualities

we experience in the physical world. (M.Coehlo,

2012). Materials have always played a major role in

our lives and the way we communicate. Walls

served historically as a visual representation surface,

clay was a means of communication, the stick

counting device used to record and document

numbers and so on.

We are witnessing, through digital technology, a

reduction of the hand to a finger. This has also

changed the way we touch and communicate with

the digital word. Our interaction with it happens in a

more discrete manner, we make contact in a more

“on and off” way.

Mark Weiser ́s insight, that computers might be

seamlessly integrated into the world, marked an

important overture in what might be called a

“material turn” (Robles and Wiberg, 2010). The

material turn in interaction design is formulated in

tangibility. Tangible interactions connect the digital

with the physical by reconsidering computation

through materiality (Robles and Wiberg, 2010).

Thanks to physical computing some artists focused

on more tangible interfaces and therefore on the use

of materials, either as a display, or as an interface.

Interactivity through materials in digital arts is a

Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art

177

characteristic that privileges the sense of touch. The

hand stops being just an extension of our body that

works in an “on and off” way; through the use of

materials in digital artworks we can now sense our

physical world in a different way, sometimes

creating a magical experience.

5 MAGICAL REALITY AND THE

EXAMPLE OF VITRAILS

5.1 Magical Reality

Professor Hiroshi Ishii and his Tangible Media

Group have a vision for the future of human-

material interaction, in which all digital information

has a physical manifestation so that we can interact

directly with it. We no longer think of designing the

interface, but rather we conceive the interface itself

as material.

The use of materials in our digital age offers the

public a new and more natural form of interaction

that brings the digital world closer to nature; a union

that can give rise to a magical reality. Subbotsky

(Subbotsky, 2010) describes the term magical reality

as a contrast to physical reality in which magical

events are forbidden. For Subbotsky magical events

may happen, at least in children's play, in fantasy,

dreams, and art. He is not interested on how magic is

achieved but what effects are experienced as magic.

For Rasmussen (Rasmussen et al, 2013)

technological artifacts can construct such magical

realities.

Hanna Landin (Landin,. 2005) investigates our

relationship to technology that could be described as

magical. She states that by the use of computer-

generated images we can create illusions, but also

through technology we can create ‘magical’ things;

things that we can in theory understand, but

somehow transcend common sense. Judith Guez

(Guez, 2015) inspired by the effects of magical

tricks, creates magical experiences through the

mixture of Illusions between the Real and the

Virtual (IRV). She uses real objects as interfaces for

the interaction with the virtual world. Her research

focuses on creating illusions between the real and

the virtual and on exploring new artistic forms that

challenge the concept of presence.

Andrea Minuto (Minuto and Pittarello, 2015) in

a workshop about the relation of smart materials, art

and technology in which students interpreted the

relation between materiality and digitality in

different modalities, noticed that when the digital

part was hidden under the hood, the visitor

experienced a “magic” interaction between different

materialities. Jongh Hepworth (Hepworth, 2007)

points out that users are used to anything happening

on their computer monitor, thus it is more interesting

to investigate magical experiences in the gray area

between virtuality and physicality. Hepworth is

describing four ingredients of magical interaction:

Surprise (something unexpected), Unordinary

(something different from a previous experience),

Unnatural (to do something you cannot do, for

example using physical objects to select virtual

content), Exciting (something you anticipate).

5.2 Magical Materiality

As all the above researches are referring mostly to

the field of design and not to the domain of arts, we

are now going to examine some artworks that use

these ingredients by combining digital technology

with materiality.

In Eye Catcher, an interactive project from Lin

Zhang and Ran Xie, a seemingly inconspicuous

frame on a wall, surprisely comes to life. Combining

receptive software and analog mechanics, it reads

your facial expression and expresses itself

accordingly, in the form of two morphing black

liquid blobs. (ferrofluid -- nanoparticles of

ferromagnetic material suspended in a carrier

liquid).

In lotus 7.0 an interactive project by Dutch

design firm studio Roosegaarde, an illuminated

reflective wall is coming to life without any

mechanical support. The self-commissioned project

is made from smart foil, lights and custom

electronics. This unordinary interaction is achieved

thanks to the smart foil which opens and closes in

response to human behavior.

In the example of interactive work of Random

International's Rain Room visitors have the

opportunity to experience what is seemingly

impossible and unnatural: the ability to control rain.

Aerial Tunes is a collaborative, tangible

interface, based on balls hovering in mid-air, which

can be manipulated individually, or collaboratively

to explore and experiment with an ambient

soundscape. Aerial Tunes exemplify how systems

can be designed to support aesthetic experiences and

promote enjoyment and excitement through a

seemingly magical and unstable display.

(Rasmussen et al., 2013).

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

178

5.3 Magical Virtuality

In Milgrams virtuality continuum (Milgram and

Kishino, 1994) as we move towards a more virtual

environment we are being more and more isolated

from the real world that is surrounding us. Our

relation with the natural environment is vanishing.

For instance, nowadays we can visit, without cost

and without spending time, many cities through

virtual reality.

In 1988 the anthropologist Alfred Gell studied

the relation of magic with technology and proposed

magic as an ideal technology; a technology that

offers to the user the best result with lesser effort.

Such technologies are being created very often the

last century but they easily lose their magic (Svanaes

and Verplank, 2000). We are used in interacting

with a device through peripherals such mouse,

keyboard, screen, HMD e.t.c but when we can

interact with a VR world through physical materials

it is quite surprising. A great example of this is

Judith Guez’s installation “Lab’surd” (le

LABoratoire de la SURvirtualité). In this work the

spectator is carried away, through the use of a virtual

reality headset, in a contemplative and evolutionary

experience of "virtual magic". Between presence and

wonder, the spectator is progressively transported

from a virtual version of his actual surroundings

towards a successive series of illusions that

transform the space. Through his journey, he is

accompanied by a small luminous sphere, one

element that he brings with him from the physical

world inside the virtual.

In her installation “Tangible Worlds”, the artist

Stela Speziali also questions the way we interact in

VR experiences. She seeks to combine the

immateriality of the digital with the sense of touch

through a tangible setup in the physical space

surrounding the user. In this example, the sense of

touch functions as a catalyst, along with visual and

auditory stimuli, for full immersion in a parallel

universe. Thus, “Tangible Worlds” questions the

perceptual correlation of what we perceive from the

outside and the inside of VR and their mutual

relationship.

In our research we envisage VR as a medium

through which we can re-invent our physical

surroundings. Magicians’ base technique to create

surprises is to establish a pattern between cause and

effect and then break it. VR immersion pattern of

wearing a headset and immediately entering to a

new world is one of the routines that we try to break

in our study (Kumari et al., 2018).

As we saw earlier, materials are used in several

cases to enrich reality with magical experiences. The

same can be done with the use of materials in a

virtual experience in order to make it more magical.

By adding materiality as a new layer between the

virtual and the real, as a “magical mirror” that is

needed to be passed in order to achieve the

immersion, we try to give to this medium another

dimension and make the experience more magical

because of a surprising, unordinary, unnatural and

exciting way of interaction – to cite Hepworth’s

ingredients - with both the physical and the virtual

world.

5.4 Case Study: VitRails

Inspired from these fields of research, the

installation VitRails, created by Nefeli

Georgakopoulou, Sofia Kourkoulakou and

Dionysios Zamplaras of the collective Continuum,

seeks to create a multisensory experience by

integrating touch into a work of mixed reality.

Interacting in a virtual environment can sometimes

by itself constitute a magical experience. When the

interactions occur through different and various

modalities, this effect can be enhanced. In VitRails,

the effort of the user to reveal and sense the virtual

environment passes by the sense of touch and the

physical materials in front of him.

Τhe installation uses a thermochromic interface.

The user, wearing a virtual reality helmet, is

immersed in a dark world, without horizon or light.

The only way to exit exists in another dimension,

this of the physical world. In the physical

environment of the user there is a painted wall with

a color that detects thermal activity. By touching and

coming in contact with it, the user can leave his

traces on this “magical interface”. These traces on

the surface become then a temporary window, which

gives him access to the virtual world.

In this work, the participant is invited to use his

body, his hands, in order to see the virtual world.

Once he achieves to access the virtual world, he has

only a few seconds to explore the events revealed on

the other side. The cracks will start to disappear

quickly, while the color of the wall returns to its

original state. In VitRails, the user is invited to be in

constant contact with the physical world around him

in order to sense and perceive the virtual. Thus, the

spectator lingers on the frontier between the real and

the virtual.

On the other side of the wall, the user – through

his temporary traces on the magical interface -

witnesses glimpses of the virtual environment. This

Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art

179

other universe consists of a room inside an

abandoned school, surrounded by the sea, which is

visible through the rear door of the room, as well as

the windows on the left wall. Outside of one of the

windows of the room the user can also see a small

boat.

Figure 1: The VR environment in VitRails.

Figure 2: Interacting with VitRails in Laval Virtual 2018.

Inside the room there are two abstract figures,

which narrate fragments of different stories. The

voices of these figures are getting louder as the user

leans on the wall. Through this setup it seems like

the visitor is placed outside of the virtual world and

in order to understand of what he is seeing he must

eavesdrop the whispers of the figures.

The room and the boat are reconstructed as a

point cloud through the use of photogrammetry. In

other words, actual photographs are used in order to

render the virtual space. In a similar way the sea and

the two beings are also rendered as animated

particles. The technique of photogrammetry is used

first of all as an aesthetic choice from the artists.

Moreover, in our case, the points of the cloud may

also be considered as the equivalent of a digital

materialand a link between the physical and the

virtual visual stimuli.

VitRails also implements bone conduction as a

means of accessing audio content. Hearing through

bone conduction occurs when the vibrations are

conducted through the skull of a listener in the inner

ear, bypassing the eardrum. This kind of devices,

called transducers, have been used in art (as well as

increasingly in consumer products) for several

decades. Despite their widespread use, because of

the “unnatural” way of producing sound through

conduction, that gives the impression that the waves

come out of the interior of objects, the transducers

continue to attract interest.

Τhe contact with the surface in VitRails also

allows one to access the vibrations coming out of

these parallel worlds. By manipulating a bone

conductor, the user is invited either to push it against

the thermal surface to make it vibrate and emit

sound, or to press it on his skull in order to acquire a

more personal and intimate experience.

Figure 3: Using the bone transducer in VitRails.

The multimodal approach suggested in VitRails

reveals the complex relationships proposed by

multisensory experiences. Sometimes, such practices

also allow us to consider a certain degree of effects

such as "sensory substitution" or “sensory

augmentation”. Sensory substitution is here

explained by the solicitation of one sense to treat the

information normally received by one of the other

senses. In our case, this occurs by the use of bone

conduction, which make one perceive sound through

vibrations on his skin and bones. This “haptic”

dimension of sound contributes to the magic of the

experience.

The sensory augmentation can be described as

the intention to extend the body's ability to sense

aspects of the environment. In VitRails the “sensory

augmentation” occurs from the fact that the user is

deprived of his vision in the beginning of the

experience. This forces him to awake his other

senses, in an effort to sense and reveal the “hidden”

aspects of the real virtual continuum presented. This

effect adds a layer to this passage from one

dimension to the other, making it more magical

through these non-ordinary and surprising modalities

of interaction with the virtual.

6 EVALUATION

The installation has been presented to the public two

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

180

times so far. The first in France in the Laval Virtual

2018 festival, in the exposition Recto VRso In,

which was entitled MatièreRéelle / MatièreVirtuelle

(3-8/04/2018). The second in Greece, during the

Athens Digital Arts Festival, in its 2018 edition

entitled Singulairty Now (24-27/05/2018).

The unordinaryand innovative way of interacting

with this VR installation led to a very high rate of

participation in both exhibitions. This was also

enhanced by the setup during the two exhibitions.

The thermochromic interface was presented as a

mere black and otherwise empty painting with a

black frame hanging inside the gallery, on a black

wall. This rather minimal setup seemed to arouse

curiosity.

The SMI, as used in VitRails, served as provider

of full metaphor (Fishkin, 2004). In the mind of the

users, the virtual wall was the same with the

physical one. According to the users, we managed a

good interaction, as leaning on the wall to see

through the cracks and hearing what is happening on

the other side connected correctly the actions

between the two worlds. The fact that the interface

was a physical material really surprised the

audience. Moreover, it even excited the part of the

public that is less interested in VR experiences,

because of its physical changes that were visible to

the non-participants. The transducer setup also

seemed toarouse curiosity to the public because of

the unnatural way of handling this small device and

their contact with the surface in order to feel the

sound vibrations.

Nevertheless, the thermochromic interface was

rather fragile, as on one hand it was influenced by

the environmental factors and on the other was

deteriorating day by day due to the use. The contact

with the bare hands of some dozens of visitors per

hour is sure to leave permanent traces or even

scratches on the surface. The temperature outside as

well as inside the gallery, or even the lighting

conditions, had an impact on the overall color of the

surface, as well as its capacity to effectively respond

to contact.

7 CONCLUSIONS AND

FUTUREWORK

While most VR installations play with vision as the

main input, VitRails invites the user to multisensory

interaction to enrich mixed reality experiences.

Through it, the user is part of a continuum between

the real world and the virtual world.

In this research, we tried to explore certain

aspects of multisensory and magical experiences in

interactive art. We think that interactive art should

challenge multiple perceptive and sensory channels

and that interaction with them should not engage our

full attention but rather immerse us in a more natural

way.

Existing research often treats the sense of touch

as a merely receptive sensation. But we have evoked

that active touch and active sensing is the way we

acquire our daily experience, the way we explore our

surroundings, whether they are physical or virtual.

The understandings of these mechanisms would be

very beneficial to haptics and human-machine

interaction in general, especially for artists, since

they should propose new ways of experiencing the

virtual.

As interaction interface design evolves towards

more tangible solutions and controllable materials,

the concept of magicality as the unordinary and

unexpected behavior of physical objects becomes

more and more fundamental. Through the use of a

technique called electrowetting Umapathi et al are

creating a graphical display that uses water droplets

to communicate information. As he puts it "The

larger idea is to provoke surprise and delight, the

way only the natural world can."

In the same way,

in the work by Sahoo et al., (2018) it is pointed out

that the lack of tangibility in digital interfaces leaves

the rich sensory capabilities of our hands and bodies

particularly under-utilised. On the contrary, adding

tangibility through materials and dynamic haptic

feedback aims to connect the physical world more

directly to our digital content, in the same way that

is imagined and highlighted by Ishii’s “tangible

bits”.

Through our research we also suggest that the

use of physical materials and the creation of tangible

interfaces creates a unique bonding of the virtual

environment with our physical dimension, thus

resulting in more magical multisensory experiences.

If the magical materiality is the human-material

interaction augmented by technology, then the

magical virtuality is the human-machine interaction

augmented by materiality.

REFERENCES

Appert, D., 2016. Conception et évaluation de techniques

d’interaction non visuelle optimisées pour de la

https://www.fastcompany.com/90169644/calm-interfaces

-are-here-and-theyre-wonderful (visited 05/01/2019)

Exploring the Virtuality Continuum Frontiers: Multisensory and Magical Experiences in Interactive Art

181

transmission d’information. Interface homme-machine

[cs.HC]. PhdThesis, Université de Toulouse,

Toulouse.

Aristote, 1993. De l’âme, GF Flammarion. Paris.

Balpe, J.-P. (Ed.), 2000. L’art et le numérique, Les cahiers

du numérique, volume 1, Hermès, Paris.

Coehlo, M., 2013. Materializing Interaction, PhD Thesis,

MIT.

Descartes, 1966. La Dioptrique, GF Flammarion. Paris.

Fishkin, K., 2004. A taxonomy for and analysis of tangible

interfaces. Personal and Ubiquitous Computing 8.

Springer-Verlag. pages 347–358.

Ginosatis, D., Kavvathas, D., Golemi, A., 2012.

Microphysics of Hardware or the Gigantism of the

Inﬁnitesimally Small. In Marshall McLuhan and

Walter J. Ong International Centenary Symposium.

SSRN Electronic Journal.

Guez, J., 2015. Illusions entre le réel et le virtuel (IRV)

comme nouvelles formes artistiques : présence et

émerveillement, Thèse, Esthétique, Science et

Technologie des Arts, Université Paris 8, Paris.

Hepworth, S., 2007. Magical experiences in interaction

design, Proceedings of the 2007 conference on

Designing pleasurable products and interfaces. ACM.

pages 108–118.

Ishii, Η., 2008. Tangible bits: beyond pixels. International

Conference of Tangible and Embedded Interaction,

TEI '08, pages xv-xxv..

Jacob, R. J., Girouard, A., Hirsheld, L. M., Horn, M. S.,

Shaer, O., Solovey, E. T. and Zigelbaum, J., 2008.

Reality-based interaction: a framework for post-wimp

interfaces. In Proc. conf. SIGCHI on Human factors in

computing systems, CHI '08, pages 201-210.

Kumari, S., Deterding, S., Kuhn,G., 2018. Why game

designers should study magic. In Proceedings of the

13th International Conference on the Foundations of

Digital Games (FDG '18). ACM.

Landin, H., 2005. Fragile and magical: materiality of

computational technology as design material.

Proceedings of the 4th decennial conference on

Critical computing: between sense and sensibility.

ACM. pages 117–120.

Levine, E., Touboul, P., 2015. Le corps. GF Flammarion.

Paris.

Loomis, J., Lederman, S. J., 1986. Tactual Perception,

Handbook of perception and human performance,

Boff, Kenneth R. (ed.), New York: John Wiley and

Sons.

Manovich, L., 2001. The language of New Media,

Leonardo, MIT Press.

Massie, P., 2013. Touching, Thinking, Being: The sense of

touch in Aristotle’s De anima and its implications,

Minerva - An Internet Journal of Philosophy, pages

74-101.

McLuhan, M., 1964. Understanding Media - The

extensions of man

, McGraw-Hill Education, Canada.

Merleau-Ponty, M., 1945. Phénoménologie de la

perception, Gallimard. Paris.

Milgram, P., Kishino, F., 1994. Augmented Reality: A

Class of Displays on the Reality-virtuality Continuum.

Telemanipulator and Telepresence Technologies.

SPIE, vol. 2351, pages 42-48.

Minuto, A. and Nijholt, A., 2013. Smart material

interfaces as a methodology for interaction: A survey

of smis’state of the art and development. In Proc. of

2nd Int. Workshop on SMIs: Another Step to a

Material Future, ACM, pages 1–6.

Minuto, A., Pittarello, F., 2015. Smart Materials: When

Art Meets Technology, In: Nijholt A. (eds) More

Playful User Interfaces. Gaming Media and Social

Effects. Springer. pages 177-196.

Prescott, T. J., et al., 2011. Active touch sensing, In

Philosophical Transactions of The Royal Society of

London, Series B, Biological Sciences

366(1581):2989-95

Rasmussen, M. K., 2013. «Magical realities in interaction

design», Proceedings of the 7th International

Conference on Tangible, Embedded and Embodied

Interaction. ACM. pages 125-128.

Rasmussen, M. K., Grönvall, E., Kinch, S., Petersen, M.,

2013. It's alive, it's magic, it's in love with you:

opportunities, challenges and open questions for

actuated interfaces. Proceedings of the 25th Australian

Computer-Human Interaction Conference:

Augmentation, Application, Innovation, Collaboration.

ACM. pages 63-72.

Robles, E., Wiberg, M., 2010. Texturing the "material

turn" in interaction design. In Proceedings of the 4th

International Conference on Tangible, Embedded, and

Embodied Interaction. ACM. pages 137-144.

Sahoo, D., Neate, T., Tokuda, Y., Pearson, J., Robinson,

S., Subramanian, S., and Jones, M., 2018. Tangible

Drops: A Visio-Tactile Display Using Actuated

Liquid-Metal Droplets. In Proceedings of the 2018

CHI Conference on Human Factors in Computing

Systems (CHI '18). ACM, New York.

Subbotsky, Ε., 2010. Magic and the Mind: Mechanisms,

Functions, and Development of Magical Thinking and

Behavior, Oxford University Press, Oxford

Svanaes, D., Verplank, W., 2000. In search of metaphors

for tangible user interfaces. In Proceedings of DARE

2000 on Designing augmented reality environments.

ACM, pages 121–129.

Varela, F., Thompson, E., Rosch, E., 1993. L’inscription

corporelle de l’esprit, éditions du Seuil. Paris.

Vertegaal, R., and Poupyrev, I., 2008. Organic User

Interfaces: Introduction to Special Issue.

Communications of the ACM, 51(6), pages 26-30.

Zerner, H., 1976. AloïsRiegl: Art, Value, and Historicism.

Daedalus, 105(1), 177-188. Retrieved from

http://www.jstor.org/stable/20024392

HUCAPP 2019 - 3rd International Conference on Human Computer Interaction Theory and Applications

182