A Hybrid Evaluation Approach for the Emotional State of

Information Systems Users

Rogério Aparecido Campanari Xavier and Vânia Paula de Almeida Neris

Sustainable and Flexible Interaction Laboratory, Department of Computing, Federal University of São Carlos,

Rod Washington Luiz, Km 235, São Carlos, Brazil

Keywords: Emotion, Users’ Emotional Experience, Emotional Evaluation, Information Systems.

Abstract: The Human-Computer Interaction community has been discussing ways to consider the user’s emotions

while interacting with computers. Emotions are a complex phenomenon, are difficult to identify and

measure and are linked to several components as cognitive aspects, subjective feelings, behavioral

tendencies, physiological responses and motor expressions. In the literature, it is possible to identify various

techniques, methods and tools for assessing the user’s emotional state. Considering the complexity of the

subject, it is necessary to combine methods to minimize the detection of false positives in the evaluation of

the user’s emotional state while interacting with information systems. This paper presents a hybrid approach

based on the emotion model described by Scherer (1984), which allows designers to check whether the

information system creates a positive, neutral or negative emotional reaction in the user. A feasibility study

was conducted in which an emotional evaluation of a web system was performed based on a group of

elderly users using tablet devices.

1 INTRODUCTION

Emotions are a key to understanding human

behavior (Cristescu, 2008). They are seen as a

mental state that arises spontaneously, without

conscious effort, and considered as feelings in

general that are often accompanied by physiological

changes such as breathing, circulation and

secretions. They are also influenced by several

external and internal stimuli including the context of

the situation, life experience, recent experiences,

personality, affect and the cognitive interpretation of

these influences (Lim et al., 2008).

Emotions are a complex phenomenon that are

difficult to identify and to measure. For

psychologists, emotions are linked to the reaction of

several components as cognitive aspects, subjective

feelings, behavioral tendencies, physiological

responses and motor expressions (Mahlke and

Mingue, 2008); (Scherer, 2005). Emotions affect our

attention, perception, memory, behavior and

cognition. Emotional responses are present in all

types of interaction between human beings, and they

lead us to quickly determine if the elements of the

environment we live in are safe or dangerous, or

good or bad (Beale and Peter, 2008); (Norman,

2004); (Piccolo et al., 2010). This type of knowledge

about emotional responses may explain why people

express their feelings while interacting with

information systems.

Therefore, it is vital to consider the users’

emotional values and their expressions during the

design process to allow the information system’s

interfaces to inspire greater confidence among users

as well as for the interface to be easier to learn and

use (Jonghwa and Andre, 2008); (Hayashi et al.,

2008). In the Human-Computer Interaction (HCI)

literature, it is possible to identify methods,

techniques and tools for the assessment of users’

emotions, (e.g., (Lefevre and Lefevre, 2005);

(Axelrod and Hone, 2008); (Cristescu, 2008);

(Yusoff and Salim, 2010). However, the separate

applications of these approaches may lead to the

detection of false positives in the identification of

the users’ emotional state due the interaction

process.

Among elderly users, we observed that in some

cases, a bad interaction experience relative to the

traditional usability metrics, such as the time of

interaction, the number of mistakes or non-

concluded tasks, whereas in other cases, the users

selected symbols indicating a good emotional

Campanari Xavier R. and de Almeida Neris V..

A Hybrid Evaluation Approach for the Emotional State of Information Systems Users.

DOI: 10.5220/0004003600450053

In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS-2012), pages 45-53

ISBN: 978-989-8565-12-9

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

experience, such as happiness or satisfaction in a

self-reported questionnaire. One user stated, “In life,

we cannot be sad” and selected a happy face in the

questionnaire. Although we agree that usability

metrics are not by themselves sufficient for one to

judge an interaction experience, in this case, the user

considered her life experience in general and did not

evaluate the emotional reaction due to her

interaction with the system. We argue that it is

necessary to combine methods, involving different

stakeholders such as users and specialists, to

minimize the detection of false positives.

This work presents a hybrid evaluation approach

that allows designers to check whether the

information system creates a positive, neutral or

negative emotional reaction in the users. Our

approach, which is based on the emotion model

described by Scherer (1984), considers a set of

methods that allows designers to identify the

emotional state of the user by considering the

subjective feelings and physiological reactions with

the user’s opinions and reactions as well as the

cognitive appraisals, behavioral tendencies and

motor expressions. A feasibility study was

conducted in which an emotional evaluation of a

web system was performed that considered a group

of elderly users using tablet devices.

This paper is organized as follows. Section 2

presents the emotion model described by Scherer

(1984). Section 3 summarizes some techniques and

tools that can be applied to evaluate each component

of the model proposed by Scherer. Section 4

describes our hybrid approach aiming to help

designers in evaluating the user’s emotional

response. Section 5 presents the feasibility study

performed to evaluate the proposed approach.

Section 6 presents a critical analysis and some of the

lessons learned. Finally, section 7 provides the

conclusion.

2 MODELS FOR EMOTIONS

The identification of human emotional states is

difficult and complex (Cristescu, 2008). Therefore,

to try to gain a better understanding of the subject,

some models describing how we feel emotions can

be found in literature. Some of these models

describe the emotions using mainly a cognitive

approach (e.g., Ortony et al., 1988), whereas others

consider multidimensional aspects as pleasure and

arousal (e.g., Osgood et al., 1975; Russell, 1983).

In this work, we have adopted Scherer’s model

(1984), which is based on components. According to

him, “it is interesting to speculate about the

possibility that specific components of emotion are

specialized to serve specific functions” (p. 297).

Scherer’s model was chosen because its approach

based on components allows us to work with each

component separately and therefore choose the

appropriate methods to evaluate the different

dimensions. Moreover, it has been successfully used

in other HCI studies to support the investigation of

emotional experiences in interactive contexts (e.g.,

(Desmet, 2003); (Mahlke and Mingue, 2008);

(Alonso et al., 2011).

Scherer’s model consists of a triangle, which is

connected to two components: cognitive appraisals

and behavioral tendencies (Scherer 1984; 2005);

(Mahlke and Mingue, 2008). The cognitive

appraisals are relevant to the assessment of the

environment including the objects and events. This

component leads to different emotions depending on

the user’s interpretation. In contrast, the behavioral

tendencies prepare the user’s emotional reactions.

According to Mahlke and Mingue (2008), these

reactions can be expressed in several ways, such as

the time required for single input operations or

completing a defined goal, the accuracy of reaching

a goal, the number of errors and the number of

creative ideas during interaction with a system.

Figure 1 illustrates Scherer’s model.

Figure 1: Scherer’s model to describe emotions (Mahlke

and Mingue, 2008).

In addition to the cognitive appraisals and

behavioral tendencies, Scherer’s model also

considers the following:

• Subjective Feelings that monitor the internal

state and the organism's interaction with the

environment, also known as conscience of emotional

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

state;

• Motor expressions that communicate reactions

and emotional and behavioral tendencies; and

• Physiological reactions that act to regulate the

system, determining the activation of

neuroendocrine processes (related to the nervous and

endocrine influences) such as heart rate, skin

conductance, blood pressure, respiration, and pupil

dilation (Shami, 2008).

Our approach considers a set of methods that allows

designers to identify the user’s subjective feelings as

reported by the user as well as cognitive appraisals

and motor expressions derived from the participation

of evaluators.

3 EMOTIONAL EVALUATION

In the literature, it is possible to identify methods,

techniques and tools with which to assess emotions

in humans. Each of them has features characterizing

them more appropriate for certain aspects of

emotions. Generally, the instruments applied in the

methods, techniques and tools can be classified as

verbal or non-verbal. In this research, we consider

an instrument as verbal when the user explicitly

verbalizes what s/he is feeling.

According to Desmet (2003), verbal instruments

enable users to express their emotion in scales (when

a user says “I am very happy” or “I am not anxious

at all”) and to report “mixed” emotions as tension.

However, they are difficult to apply across cultures

because it may not be easy to translate emotions into

words. On the other hand, non-verbal instruments

can be considered discreet and independent of

culture and language. However, they can be

subjective as they generally use universal symbols

such as pictograms.

Figure 2 presents a taxonomy, which classifies

emotional assessment metrics as verbal or non-

verbal. Each final node in the taxonomy represents

one of the five components of Scherer’s model, and

each parent node represents a set of available

methods, techniques, tools and instruments that can

be used to measure a component.

Cognitive assessments are linked to the

interpretation of a situation and further development

of emotions. This component can be measured by

the Geneva Appraisal Questionnaire (GAF) (Geneva

Emotion Research Group, 2010), the Think-aloud

method (Someren et al., 1994) and the Subjective

Discourse Analysis (Lefevre and Lefevre, 2005).

Although the Subjective Discourse Analysis

considers spoken statements, this technique was

classified as non-verbal because users do not

explicitly say what they are feeling. The evaluators

should interpret the statements spoken during the

user’s interaction and classify the related emotion.

According to Scherer (2005) and Desmet (2003),

there is no objective method capable of measuring

subjective feelings. It is necessary to query the user,

and thus, the methods involve self-assessment. Most

of the methods for evaluating subjective feelings are

non-verbal, such as the SAM (Self-Assessment

Manikin) (Lang, 1985), Emocards (Reijneveld et al.,

2003), and Preemo (Desmet, 2003). A verbal

instrument is the Affect Grid (Russell, 1989).

The motor expressions are related to facial

movements, body gestures as well as to some

characteristics of speech as speed, intensity, melody

and sound. Methods that can be applied include the

Facial Action Coding System (FACS) (Ekman et al.,

2002), the Ten Heuristics of Emotion (Lera and

Domingo, 2007) and electromyography.

Physiological reactions are non-verbal and can

be measured by electrocardiogram, respiration rate,

electrodermal activity, electromyography,

pupillometry, etc. Physiological reactions allow

designers to evaluate the user’s emotional responses

in an experimental context once the users

spontaneously and unconsciously reveal their

emotions (Cristescu, 2008); (Yusoff and Salim,

2010). However, most of these evaluations require

expensive instruments and are intrusive and complex

(Axelrod and Hone, 2008); (Cristescu, 2008).

Finally, behavioral tendencies are also non-

verbal and generally are evaluated by performance

indicators, such as the time required to complete a

task, the accuracy of reaching a goal, the number of

errors and the number of creative’ ideas during the

interaction (Mahlke and Mingue, 2008).

4 A HYBRID APPROACH

Aiming to minimize the detection of false positives

in the emotional evaluation of information systems,

this work proposes a hybrid approach based on the

emotion model described by Scherer (1984).

Considering the evaluation methods and instruments

presented in the literature, we have selected a subset

that matches the five components of the Scherer’s

model. This selection considered the main

stakeholder (user or specialist) responsible for the

final result of each method or instrument, aiming to

balance the final emotional assessment of the

information system.

AHybridEvaluationApproachfortheEmotionalStateofInformationSystemsUsers

Figure 2: A taxonomy for emotional evaluation methods, techniques and tools.

The subjective feelings demand a self-report

instrument, and we have selected the SAM (Lang,

1985) to measure it. The SAM is composed of three

sets of figures that represent pleasure, arousal and

dominance. Each dimension is evaluated using a

scale ranging from 1-9 in which the user selects one

circle. As a pictogram, it reduces the cultural

differences, and as a non-verbal instrument, it avoids

problems with the verbalization of emotions.

Moreover, it considers the dominance aspect

explicitly.

In our approach, we suggest that SAM be applied

after the interaction with the system. The evaluator

presents the instrument to the user and asks him/her

to classify his/her experience by choosing one of the

nine circles in each dimension (pleasure, arousal and

dominance). Figure 3 illustrates the pictograms

adopted by SAM for the pleasure dimension. The

emotional response is considered positive if the user

selects one of the circles indicated by the V+. The

negative experience options are represented by V-,

and the neutral experience is the central option and

represented by VN. The final result can be reached

for each dimension by adding the number of votes of

all users in V+, VN and V-.

Figure 3: Pictograms adopted by the SAM questionnaire

for measuring pleasure and the emotional values scales.

Following Mahlke and Mingue (2008), we have

adopted effective and efficient metrics to measure

the behavioral tendencies. They include the time

required to complete a task and the number of errors

or help requests, among others. Designers may not

have difficulty collecting these metrics and

evaluating if the final result is positive, negative or

neutral, as they are commonly measured in

traditional usability tests.

The physiological reactions can be assessed by

sensors because they are related to neuroendocrine

processes. The data should be collected during the

interaction instead of only at the end of interaction.

Moreover, the sensor should not disturb the user

during the interaction. The results should be

compared to baseline values established by the

designers. If heart rate is collected, a baseline value

could be 85 beats per minute. With a baseline value,

designers can evaluate if the final result is positive,

negative or neutral. However, sensors that can be

used during the interaction, which save the data and

do not disturb the user, are generally expensive.

The motor expressions component is assessed in

our approach by the Ten Emotion Heuristics (Lera

and Domingo, 2007), which are frowning, raising

eyebrows, looking from a distance, smiling,

compressing the lips, moving his mouth, vocal

expressions, hand touching the face, going back to

the chair and leaning the trunk forward. The

evaluation is divided into two steps. In the first one,

a group of pre-selected appraisers watch videos of

the user’s interaction. The videos can be captured by

common webcams and should record the user’s face

and body.

For each heuristic identified, we recommend that

the appraisers register the time it occurred, the task

the user was doing, the heuristic or set of heuristics

identified and a description of the emotional aspects.

In the second step, the evaluators meet, discuss and

build a final list containing the heuristics found.

According to Lera and Domingo (2007), the final

emotional experience evaluation is set as negative if

five or more negative heuristics are found per user.

Because it is a heuristic evaluation, the collective

common sense can identify the emotional experience

of a group of users more accurately than an

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

Figure 4: A hybrid approach for assessing the emotional state of users.

individual appraiser. Furthermore, facial recognition

software is generally expensive, and FACS demands

an experienced assessor to apply it properly.

The cognitive appraisals are measured by an

adaptation of the Subjective Discourse Analysis

(Lefevre and Lefevre, 2005). Using the same video

as that recorded for the heuristic evaluation, the

assessor lists key-expressions that were spoken

spontaneously by the users during the interaction.

Key expressions are central ideas that represent a

synthesis of the discursive content, such as "And

now?" or "Should I click here?”. In addition to the

expression, the assessor should add a description of

the emotional situation in which that phrase was

said, for example, in a moment of confusion, joy, or

surprise. Thus, based on the description, the

evaluator classifies the expression in terms of

positive, neutral or negative.

After analyzing the videos and selecting the key

expressions for each user, a final list of key

expressions should be created. To be selected for the

final list, a key expression should be used by more

than one user. Finally, considering the most

frequently spoken expressions and their

classification as positive, negative or neutral, the

designer can define the users’ emotional response to

that system considering the cognitive appraisals.

Figure 4 shows the hybrid approach for assessing the

emotional state of the users.

Applying the methods as described here, the

designer has partial and complementary emotional

responses of the users to the interaction with the

information system. Considering the evaluation of

each component, it is possible to reach a more

comprehensive result and to decide if the

information system creates a positive, neutral or

negative emotional reaction in the users.

Table 1: Stakeholders and the final decision on the user’s

emotional response.

Scherer’s

component

Method Stakeholder

Subjective

feelings

SAM User

Behavioral

tendencies

Effective and

efficient metrics

Designer

Physiological

reactions

Sensors User

Motor

expressions

Ten Heuristics of

Emotion

Group of designers

Cognitive

appraisals

Subjective

Discourse

Analysis

Designer

Moreover, the results from each component

support designers altering different aspects of the

interface. Considering the most recurrent heuristic,

for instance, it is possible to learn if the users are

frustrated or confused. By SAM, it is possible to see

if the users are excited but not confident, and by the

sensor results, it is possible to see if the users are

anxious. Finally, Table 1 presents the methods that

are part of our approach and the stakeholder who

makes the final decision regarding the user’s

AHybridEvaluationApproachfortheEmotionalStateofInformationSystemsUsers

emotional response. We argue that these

complementary views minimize the detection of

false positives because we consider not only the

information provided by users but also the designers'

opinions to classify the emotional experiences.

The next section presents a feasibility study

applying the proposed approach.

5 FEASIBILITY STUDY

Aiming to assess the feasibility of our hybrid

approach, we selected a group of seven elderly users

to evaluate a website about food and recipes using

tablet devices. The evaluation occurred in the Social

Reference Assistance Center (CRAS, in its

Portuguese acronym) in São Carlos-SP, Brazil. The

users were selected according to their age, education

level and experience with devices and were asked to

find a specific recipe, starting from the website

home page. As the elderly users were not

accustomed to using to tablet devices, one of the

researchers acted as an active moderator, answering

questions when the users asked for help.

Each user was filmed by two common cameras.

One camera was focused on the user's body, and the

other was registering the user’s interaction with the

device. After each interaction, the users were asked

to fill in the SAM questionnaire. The users’

selections were added in each dimension, and Table

2 summarizes the results. The pleasure dimension

had a positive assessment with seven positive votes

(V+). The arousal dimension was also ranked as

positive with six positive votes (V+). On the other

hand, the dominance dimension had a negative

assessment with six negative votes (V-). The users

reported that they had a pleasurable and exciting

interaction but that they were not in the control of it.

Table 2: Results of the SAM assessment.

Pleasure Arousal Dominance

V- VN V+ V- VN V+ V- VN V+

0 0 7 0 1 6 6 0 1

After the interaction experience, we applied the

Ten Heuristics of Emotion method. The evaluation

was performed by a group of six evaluators,

including five from the computer science field and a

professional nurse. The nurse was invited to join the

group based on the idea that a professional from the

health area could provide a complementary view in

an evaluation with elderly users. One of the

evaluators had experience in applying the method,

and the others received one-hour of training. The

final video, with all of the users, was 53 minutes and

41 seconds long. Figure 5 illustrates three different

moments in the video and the heuristics identified.

Figure 5: Three examples of emotional heuristics.

In this feasibility study, three users had a positive

experience with less than three different negative

heuristics identified, and four had a negative

experience with five or more negative heuristics

identified. With a small number of users and a non-

expressive difference in the final result, we

classified the assessment of motor expressions as

neutral.

The Subjective Discourse Analysis, as described

in our approach, was applied by the researcher that

acted as the moderator during the interaction.

Analyzing the same video used to evaluate the

heuristics, it was possible to identify the key

statements made by the users in addition to their

interaction context. The selected statements were

classified as positive, neutral or negative.

Furthermore, the most frequently spoken key

statements were considered in the final cognitive

appraisals evaluation. Table 3 summarizes the data

collected.

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

Table 3: The most frequently spoken key expressions.

Key expressions Interaction context Experience

Number

of users

Here?

Demonstrate doubt

or do not know how

to proceed in the

task.

Negative 5

Recipes!

Realize where to

click to accomplish

the task.

Positive 4

And now?

Perform an action

and do not know

what will happen. It

was also said in

moments of

frustration or when

the user was

confused.

Negative 4

I do not know.

Difficulty in

understanding how

the system works,

how to interact, or

when abandoning the

activity.

Negative 3

I do not

understand.

Feelings of

confusion or

disorientation due to

do not knowing how

the information was

presented in the

interface.

Negative 3

Checking the list of the top five spoken

expressions, i.e., those most frequently stated by the

users, we evaluated the cognitive appraisal as

negative. The top five statements were chosen, and

these statements were sufficient to indicate an

emotional response tendency. If a non-clear

tendency is reached in a study, designers should

consider additional frequently spoken statements.

The behavioral tendencies were evaluated

considering the number of times the users asked for

help while interacting. As the expected interaction

time was short (approximately two minutes), we

considered that up to two solicitations for help

would be classified as positive. Three solicitations

for help were classified as neutral, and more than 3

solicitations were classified as negative. Two users

had a positive behavioral tendency, and one user’s

tendency was classified as neutral. Four users asked

for help more than three times, and their behavioral

tendencies were classified as negative. The final

result for this component was deemed negative.

The physiological measures were not evaluated

in this feasibility study because of the high cost of

the specific equipment and sensors required. Table 4

presents the final results of the website emotional

evaluation.

Table 4: Results of the emotional assessment to the

cooking site using the hybrid approach.

Scherer’s

component

Method Evaluation

Subjective feelings

SAM- dimension of pleasure Positive

SAM- dimension of arousal Positive

SAM- dimension of dominance Negative

Behavioral

tendencies

Effective and efficient metrics Negative

Physiological

reactions

Sensors Not applied

Motor expressions Ten Heuristics of Emotion Neutral

Cognitive

appraisals

Subjective Discourse Analysis Negative

Thus, an analysis of the results of each

evaluation revealed that the emotional state of users

while interacting with the website was classified as

negative.

6 CRITICAL ANALYSIS AND

LESSONS LEARNED

Using this hybrid approach, we were able to identify

a set of relevant information about the emotional

experience of users. For instance, applying the SAM

questionnaire, it was possible to realize that even

when elderly users do not have control over

technology, the interaction can be pleasurable and

excited. Other lessons learned include the following:

• Even in an evaluation of a user group with

similar profile characteristics, there were variations

in the users’ emotional states.

• The users who were not familiar with the menus

and search engines triggered a greater number of

negative heuristics. This finding suggests a bad

relationship between less experience and the

emotional response, i.e., the less the user knows

about the system interaction logic, the more negative

is the emotional response to the interaction.

• The use of the Subjective Discourse Analysis

allowed us to observe that some key statements are

made by most of the users in similar interaction

experiences. This observation suggests that affective

systems could recognize these statements and

change their user interfaces based on their

occurrences.

• The evaluators who applied the Ten Heuristics of

Emotion noted that the process of evaluating the

heuristics requires a significant amount of time.

"The method is inexpensive and fairly simple to run;

however, it is difficult to achieve due to the large

physical and cognitive effort required by the

assessor", said one of the evaluators. Therefore, the

AHybridEvaluationApproachfortheEmotionalStateofInformationSystemsUsers

emotional evaluation performed used limited

resources, but demanded time, especially of inexpert

evaluators.

Finally, the proposed approach supports the

evaluation of the users’ emotional responses due to

the interaction with information systems. The

evaluation led to a final assessment of the system.

However, the methods could also be applied to

assess each user’s emotional state. Therefore,

through flexible and more accessible design

solutions, users with low literacy levels and less

experience with technology could have better

emotional experiences.

7 CONCLUSIONS

This paper presented a hybrid approach to the

emotional evaluation of information systems. The

approach is based on Scherer’s model (1984), and

the evaluation methods were selected and adapted to

measure the user’s emotional response for the five

components. A feasibility study was conducted

considering a group of elderly users using tablet

devices. The results suggest that the proposed

approach can be easily applied, and moreover, it is

relatively inexpensive.

The feasibility study also suggested that the

approach is able to evaluate the users’ emotional

responses to the interaction, considering the software

and the hardware used. In the elderly users’ cases,

less experience with tablets certainly influenced the

final result. Moreover, the methods used also

allowed the evaluation of a web information system.

Future research can be performed to determine a

better combination of methods for specific system

domains. As Yusoff and Salim (2010) noted for

games, for instance, there are various physiological

features that can be measured and related to the

emotional response.

Future work will consider applying this hybrid

approach to identify the emotional state of young

people and adults during the interaction with

information systems in order to analyze the degree

of emotional experience obtained between and

among groups of users. Therefore, identifying the

users’ emotional states, we intend to improve the

design solutions to create flexible interfaces that

focus on satisfaction and emotional aspects of these

users.

ACKNOWLEDGEMENTS

This work was funded by CAPES - Brazil. The

authors thank colleagues from LIFeS-UFSCar for

their insightful comments. The authors especially

thank the elderly users, who gladly received the

research group and participated in the feasibility

study, the designers and the referees for their

contributions.

REFERENCES

Alonso, M. B., Hummels, C. C. M., Keyson, D. V. and

Hekkert, P. P. M., 2011. Measuring and adapting

behavior during product interaction to influence affect.

Personal and Ubiquitous Computing.

Axelrod, L. and Hone. K., 2008. Measuring affect:

Differentiating positive affect and politeness. CHI

2008 Workshop.

Beale, R. and Peter, C., 2008. The Role of Affect and

Emotion. Affect and Emotion in Human-Computer

Interaction, 4868, pp.1-11.

Cristescu, I., 2008. Emotions in human-computer

interaction: the role of nonverbal behavior in

interactive systems. Revista Informática Econômica,

2(46), pp.110-116.

Desmet, P. M. A., 2003. Measuring Emotions:

Development and application of an instrument to

measure emotional responses to products. M.A. Blythe,

A. F. Monk, K. Overbeeke, & P. C. Wright (Eds.),

Funology: from usability to enjoyment, pp.111-125.

Ekman, P., Friesen, W. V. and Hager, J. C., 2002. The

Facial Action Coding Systems, 2rd ed, Salt Lake City:

Research Nexus eBook. London: Weidenfeld &

Nicolson (world).

GAQ, 2010. Geneva Appraisal Questionnaire Format,

development, and utilization. [online]. Available

<http://www.affectivesciences.org/system

/files/page/2636/GAQ_English.PDF> [Accessed 25

January 2012]

Hayashi, E. C. S. et al., 2008. Avaliando a qualidade

afetiva de sistemas computacionais interativos no

cenário brasileiro. IHC 2008 - VIII Simpósio

Brasileiro de Fatores Humanos em Sistemas

Computacionais pp.1-5.

Jonghwa. K. and Andre, E., 2008. Emotion Recognition

Based on Physiological Changes in Music Listening.

IEEE Transactions on Pattern Analysis and Machine

Intelligence, 30(12), pp.2067 – 2083.

Lang, P. J., 1985. The cognitive psychophysiology of

emotion: Fear and anxiety. Hillsdale, NJ: Lawrence

Erlbaum, pp. 131-170.

Lefèvre, F. and Lefèvre, A. M. C., 2005. Discurso do

sujeito coletivo - Um novo enfoque em pesquisa

qualitativa (Desdobramentos). Livraria Resposta.

Lera, E. and Domingo, M. G., 2007. Ten Emotion

Heuristics: Guidelines for assessing the user’s

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

affective dimension easily and cost-effectively. BCS-

HCI '07 Proceedings of the 21st British HCI Group

Annual Conference on People and Computers:

HCI...but not as we know it, 2, Publisher British

Computer Society Swinton, pp.163-166.

Lim, Y. et al., 2008. Emotional Experience and Interaction

Design. Affect and Emotion. Human-Computer

Interaction, 4868, pp.116-129.

Mahlke, S. and Mingue, M., 2008. Consideration of

Multiple Components of Emotions in Human-

Technology Interaction. Affect and Emotion in

Human-Computer Interaction, 4868, pp.51-62.

Norman, D.A., 2004. Why We Love (or Hate) Everyday

Things. Basic Books.

Ortony, A., Clore, G.L. and Collins, A., 1988. The

cognitive structure of emotions. Cambridge: Press

Syndicate of University of Cambridge.

Osgood, C. E, May, W. H. and Miron, M. S., 1975. Cross-

Cultural Universals of Affective Meaning. University

of Illinois Press; 1ed.

Piccolo, L. S. G., Hayashi, E. C. S. and Baranauskas, M.

C. C., 2010. The Evaluation of Affective Quality in

Social Software: Preliminary Thoughts. II WAIHCWS,

IHC2010, pp.29-38.

Reijneveld, K., Looze, M., Krause, F. and Desmet P.,

2003. Measuring the Emotions Elicited by Office

Chairs. DPPI '03 Proceedings of the 2003

international conference on Designing pleasurable

products and interface, pp.6 - 10.

Russell, J.A., 1983. Pancultural Aspects of the Human

Conceptual Organization of Emotions. Journal of

Personality and Social Psychology, 45(6), pp.1281-

1288.

Russell, J. A., 1989. Affect Grid: A Single-Item Scale of

Pleasure and Arousal, Journal of Personality and

Social Psychology, 57(3), pp.493-502.

Scherer, K. R., 1984. On the Nature and Function of

Emotion: A Component Process Approach. K.R.

Scherer & P.Ekman Approaches to Emotion.

Hillsdale, N.J.: Lawrence Erlbaum, 293-317.

Scherer, K. R., 2005. What are emotions? And how can

they be measured? Social Science Information & 2005

SAGE Publications, 44(4), pp.695-729.

Shami. N. S. et. al., 2008. Measuring Affect in HCI:

Going Beyond the Individual. CHI EA '08 CHI '08

extended abstracts on Human factors in computing

systems.

Someren, M. W., Barnard, Y. F. and Sandberg, J. A. C.,

1994. The Think Aloud Method: A practical guide to

modeling cognitive processes. London: Academic

Press.

Yusoff, N. M. and Salim, S. S., 2010. SCOUT and

Affective Interaction Design: Evaluating Physiological

Signals for Usability in Emotional Processing.

Computer Engineering and Technology (ICCET),

2010 2nd International Conference on, pp.201-205.

AHybridEvaluationApproachfortheEmotionalStateofInformationSystemsUsers