AffectPhone: A Handset Device to Present User’s

Emotional State with Warmth/Coolness

Ken Iwasaki

, Takashi Miyaki

and Jun Rekimoto

2,3

Graduate School of Interdisciplinary Information Studies, The University of Tokyo

7-3-1 Hongo, Bunkyo-ku Tokyo, Japan

Interfaculty Initiative in Information Studies, The University of Tokyo

7-3-1 Hongo, Bunkyo-ku Tokyo, Japan

Sony Computer Science Laboratory, 3-14-3 Gotanda, Shinagawa-ku Tokyo, Japan

Abstract. We developed AffectPhone, a system that detects a user’s emotional

state using the GSR, and conveys this state via changes in the temperature (wamrth

or coolness) of the back panel of the other handset. Since GSR is a good measure

of a user’s level of arousal, we detect the GSR using electrodes attached to the

sides of the handset. When the user’s level of arousal increases or decreases, a

Peltier module in the back panel of the other device generates warmth or cool-

ness. This system does not require special sensors to be attached to the user’s

body, and therefore, it does not interrupt the user’s daily use of the mobile phone.

Moreover, this system is designed to convey non-verbal information in an ambi-

ent manner, and therefore, it would be more efﬁcient than displays or speakers.

This system is expected to help enhance existing telecommunication.

1 Introduction

During face-to-face communication, non-verbal cues convey as much information as do

the verbal ones.[1] However,when using a computer for communication,it is difﬁcult to

convey non-verbal information. We believe that conveying such non-verbal information

would improve existing telecommunication, and therefore, we have developed a system

to facilitate the same.

1.1 Non-verbal Communication Channel

We aimed to develop a system that provides non-verbal communication channel in ad-

dition to the existing telecommunication. A communication channel generally includes

the information to be output and a sense as the input. For example, when we commu-

nicate over a telephone, our voice is the output, and auditory sense is the input (Figure

1(a)). In this system, we selected the galvanic skin response (GSR) as the output, and

temperature sensation as the input (Figure 1(b)).

Iwasaki K., Miyaki T. and Rekimoto J. (2010).

AffectPhone: A Handset Device to Present User’s Emotional State with Warmth/Coolness.

In Proceedings of the 1st International Workshop on Bio-inspired Human-Machine Interfaces and Healthcare Applications, pages 83-88

DOI: 10.5220/0002839200830088

 SciTePress

Fig.1. Communication channels. (a) Normal telephone channel (Voice and Hearing). (b) Pro-

posed non-verbal communication channel (Arousal and Temperature).

1.2 Selection of Input and Output

Humans use many non-verbal cues in their day-to-day communication. These cues in-

clude apparent ones such as body language, tone of voice, and facial expressions. How-

ever, we focus on non-verbal information that is not apparent, such as physiological

information in this study.

We selected GSR as the output information, because Picard et al. suggested that

GSR is a good measure of a user’s arousal[2]. Moreover, the GSR can be acquired by

using only two electrodes on the user’s ﬁngertip, and no special sensors need to be

attached to the user’s body.

We selected temperature sensation as the output, because warmth or coolness is

often indicative of ones emotions, as borne out by expressions such as, ”he is a cold

man” or ”a heart-warming story.” Moreover, such haptic feedback does not interfere

with user’s hearing or sight. In other words, the user can convey his/her emotional state

in an ambient manner.

2 Related Work

Wang et al. used physiological sensors and animated text to develop a system that helps

in communicating emotions in an online chat.[3] This system detects a user’s level of

arousal from the GSR, and presents the user’s emotional state as animated text. This

system uses GSR as non-verbal information, but it also presents it as an attribute of the

text. The objective of this system is to make text chat more efﬁcient; however, verbal

information is mainly used in text chats. Therefore, we focus on telephonic communi-

cation.

Brave et al. proposed inTouch, a medium for haptic interpersonal communica-

tion.[4] inTouchintroduceshaptic sensations in interpersonalcommunicationand serves

to enrich the communication. However, haptic sensations cannot be generated unless a

user voluntarily moves the device, and therefore, this method does not suitably mimic

typical day-to-day interactions. In contrast, our system can determine a user’s emotional

state from physiological information that the user cannot control voluntarily. Moreover,

our system is integrated in a mobile phone we usually use, and therefore, it does not

interrupt a user’s daily use of the phone.

Vaucelle et al. designed haptic interfaces for therapeutic purposes.[5] A part of their

research, called the Cool Me Down project uses temperature sensations for therapy.

The device developed by them can be discretely worn by the user and only activated

when necessary; this would help patients self-administer soothing sensory grounding

treatment. Their paper discusses the design concepts of haptic interfaces for therapeu-

tic purposes, but not for communication. In contrast, we focus on enhancing existing

telecommunication.

Fig.2. AffectPhone. (a) Front and rear view of the system. (b) GSR window.

3 AffectPhone

3.1 Concept

GSR has been stated to be a good measure of a user’s level of arousal, which indicates

the user’s level of excitement. We selected temperature sensation as the input because

temperature changes are generally reﬂective of ones emotions. In this system, a user

can feel changes in the emotional state of the person he/she is conversing with in the

following manner:

– When a user’s level of arousal increases, the system makes the back panel of the

other device warm.

– When a user’s level of arousal gradually decreases for approximately 30 seconds,

the system makes the back panel of the other device cool.

3.2 System Conﬁguration

AffectPhone consists of two electrodes for detecting the GSR and a Peltier module for

providing informationon temperature change (Figure2(a)). This system provides a non-

verbal communication channel in addition to the existing telecommunication channel in

a normal mobile phone. In other words, a user can convey his/her level of arousal to the

person he/she is conversing with in an ambient manner. The user’s GSR can be detected

from the two ﬁngers in contact with the phone, and temperature changes can be sensed

by the palm. This system does not require special sensors to be attached to the users

body, and therefore, it does not interrupt the user’s daily use of the mobile phone.

AffectPhone is designed to convey a users arousal in an ambient manner. Moreover,

the user can be aware of the arousal level of the person he/she is talking to, by viewing

the GSRs in the GSR window (Figure 2(b)).

3.3 Feasibility Test

In order to study the feasibility of AffectPhone, we conducted preliminary experiments.

We made a subject listen to music for approximately 4 minutes, and acquired the GSR

signal using AffectPhone.

Figure 3 shows the GSR signal acquired using AffectPhone. The sampling rate was

5 Hz. Because the skin resistance decreases with an increase in the level of arousal, we

calculated a user’s level of arousal (a(t)) as follows:

a(t) = 1000 − R (1)

where R is resistance of the skin. (k Ohm)

A GSR signal consists of three elements—rise time, amplitude, and half recovery

time (Figure 4). As shown in Figure 3 and Figure 4, the rise time, amplitude, and half

recovery time are 5-10 s, 60-90, and 20-25 s, respectively. In comparison to Gasperi’s

web cite[6], we concluded that this system successfully acquires GSR signal.

Fig.3. GSR signal acquired from AffectPhone. Peaks indicated by arrows indicate sudden in-

crease in the level of arousal.

3.4 GSR Signal Analysis

The above-mentioned test conﬁrms that the proposed system can be used to acquire

GSR signals. Here, we discuss how to analyze the acquired GSR signals.

The amplitude of the GSR signal indicates a change in the user’s emotional state

(get angry, happy, etc.). In such a situation, the system makes the back panel of the

other device warm when the following condition is satisﬁed.

da/dt > d

(2)

Fig.4. Period from 120 s to 220 s shown in Figure 3. We considered the gradual decrease after

the peak to correspond to a decrease in the level of arousal.

where a denotes the user’s level of arousal, t, the time in seconds, and d

, the threshold

of difference, respectively.

After the half recovery time, the system turns off the Peltier module of the other

device.

a(t) = (a

− a

)/2 (3)

where a

denotes the level of arousal in the amplitude and a

, the basal level of arousal.

When a user’s GSR gradually decreases after recovery, we consider that the user’s

level of arousal has decreased (Figure 4). In such a situation, the system makes the back

panel of the other device cool when the following condition is satisﬁed:

a(t) − a

< 0 (t

< t < t

+ 30) (4)

where a

denotes the basal level of arousal, t

, the time of recovery.

4 Potential Applications of AffectPhone

AffectPhone can convey a user’s level of arousal in terms of changes in the temperature

(warmth or coolness) of the back panel of a phone. This system is useful not only when

a user is talking on the phone, but also when the phone is ringing. When the phone rings,

the system determines the level of arousal of the caller. If this system ﬁnds widespread

use, the following scenario might become possible. If the caller is upset, the receiver can

be made aware of the same even before answering the phone. A user can detect whether

or not a call is important, and thus deal with a difﬁcult situation more effectively.

Fig.5. Potential applications of AffectPhone

5 Conclusions and Discussion

In this study, we designed AffectPhone, a handset device that can be used to identify a

user’s emotional state via temperature changes (warmth or coolness). This device has

been designed to provide a non-verbal communication channel in addition to the exist-

ing telecommunication channel in a normal mobile phone. However, further improve-

ments must be made to this system. In the future, we intend to focus on user evaluations

of factors such as the following:

– The extent to which a user’s level of arousal changes when talking on the phone.

– The accuracy of determination of temperature changes by the user.

Acknowledgements. We are grateful for being adopted for part of this project as ”MI-

TOH youth” project in Information-Technology Promotion Agency in Japan.

References

1. Vargas, M. F.: Louder Than Words: An Introduction to Nonverbal Communication. Iowa State

Press (1986).

2. Picard, R. W.: Affective Computing. MIT Press (2000).

3. Wang, H., Prendinger, H., and Igarashi, T.: Communicating Emotions in Online Chat Using

Physiological Sensors and Animated Text. Proc. of ACM SIGCHI Conference on Human

Factors in Computing Systems.(2004)

4. Brave S., and Dahley A.: inTouch: a medium for haptic interpersonal communication. Proc.

of ACM SIGCHI Conference on Human Factors in Computing Systems.(1997)

5. Vaucelle C., Bonanni L., and Ishii H.: Design of Haptic Interface for Therapy. Proc. of ACM

SIGCHI Conference on Human Factors in Computing Systems.(2009)

6. Gasperi M.: Homemade Galvanic Skin Response Meter http://www.extremenxt.com/gsr.htm