USING SENSORS TO DETECT STUDENT’S EMOTION

IN ADAPTIVE LEARNING ENVIRONMENT

Hippokratis Apostolidis and Thrasyvoulos Tsiatsos

Computer Science Department, A.U.TH University, Ag. Dimitriou, Thessaloniki, Greece

Keywords: Bio-signals, Biofeedback, GSR, Emotion detection, CSCL, Arduino.

Abstract: The purpose of this paper is a case study of the development of a device that could collect and evaluate bio-

signals by people engaged in learning activities. The physiological reactions of the body, affects the

conductivity of the skin producing bio-signals that provides the opportunity to estimate some basic human

emotional states. In particular stressful situations have resulted in increased moisture in human skin,

reducing the resistance and increasing the conductivity of the skin to electrical current. In this case study

there is a reference to adaptive collaborative learning support. This work suggests that emotional state

regulation may be an important factor in implementing the adaptivity of learning activities.

1 ADAPTIVE COLLABORATIVE

LEARNING SUPPORT

Adaptive support, providing assistance to student

when and where it is needed, might improve or

make more suitable many fixed forms of support

(Rummel and Weinberger, 2008), and has been

shown a more positive effect on student learning

(Kumar, et al. 2007). Nevertheless it is difficult to

design an ACLS (adaptive collaborative learning

support) system due to the fact that it is very

complicated and it contains several adaptive rules

and parameters that must be taken into account. The

aim of this work is to support a collaborative

learning activity.

To achieve comprehensive information about the

people who are collaborating there is a need to

collect information about a wide range of features

that affect the quality of collaboration. An important

aspect might be the mood of team members during

their collaboration. Taking into consideration all the

necessary features of a collaborative scenario, the

objective is to regulate, to differentiate and resolve

various issues and problems that arise during

collaborative activity in order to convert students to

active learners, increasing interaction and mainly

increase confidence levels of students to develop

cognitive skills. Therefore an adaptation pattern can

function as a scaffolding mechanism to regulate the

educational procedure.

2 STUDENT EMOTIONS

Emotions are very important functions that affect

students' academic motivation, behaviour,

performance, health, and development of their

personality.

Many researches started in 1950 (Zeidner, 1998)

have considered with the anxiety of students during

tests and produced sufficient knowledge that can

inform the educational practice. However, apart

from the stress, there is little research on the other

emotions of students, since it is difficult to draw

firm conclusions about the feelings experienced by

most students (Schutz and Pekrun, 2007).

Students are getting confused when faced with

contradictions and misunderstandings (Festinger,

1957; Graesser, et al. 2003; Graesser and Olde,

2005; Piaget, 1952). They are also getting

disappointed by the difficulties that may appear

during the learning activity (Dweek, 2002; Stein and

Hernandez, 2007.

The influence of emotions during problem

solving, and participation in educational activities is

very important because it can affect positively or

negatively the learning process (Allen and Carifio,

1995). Emotions are very relevant to cognitive

function and thus play a key role in all phases of

problem solving, influencing the representation and

the reception of information. According to modern

emotion theories, the strong or weak emotion

arousal depends on how great a difficulty is and the

Tsiatsos T. and Apostolidis H. (2011).

USING SENSORS TO DETECT STUDENT’S EMOTION IN ADAPTIVE LEARNING ENVIRONMENT.

In Proceedings of the Second International Conference on Innovative Developments in ICT, pages 60-65

DOI: 10.5220/0004471700600065

 SciTePress

valence, positive or negative,

depends on how the

person evaluates the inconvenience that has been

appeared to him (Mandler, 1984b; Lazarus, 1991).

For example, if someone is stuck and she/he is

unable to bypass a difficulty, then the valence may

be interpreted as highly negative (Dweek, 2002).

There is no comprehensive theory that deals

extensively with the emotions (e.g., confusion,

frustration, etc.) that may occur during educational

activities, and considering how they affect

performance. There is some evidence that positive

and negative emotional experiences play a role

during problem-solving. For example, flexibility,

creative thinking and effective way of making

decisions are connected with experiences that have a

positive effect (Fielder, 2001; Isen, et al. 1987; Isen,

2001).

The student class is an emotional space. The fact

that learning and achievement is critical to students’

educational development means that the academic

activities often provoke strong emotions.

2.1 Students’ Emotional Effects in

Learning Activities

The experimental research showed that the mood

and the emotions facilitate the processes of working

memory, such as positive information relating to the

person is stored in long term memory and it is

retrieved more easily when he or she is in positive

mood. On the contrary this process is getting very

hard when this person is in a negative mood, and

even worse if the information is negative (Olafson

and Ferraro, 2001). This suggests that positive mood

can enhance students' motivation to attend actively

learning processes, while a negative mood can

trigger avoidance tendencies. It is important to

consider that creative, flexible, and holistic ways of

thinking are facilitated by a positive mood, while a

negative feeling enhance complexity and poor

creativity and flexibility (Lewis and Haviland-

Jones, 2000). Most of these studies have focused on

students’ anxiety during the test (Zeidner, 1998,

2007). Research on stress during tests showed that

this feeling reduces the performance in complex or

difficult learning tasks that require cognitive

resources (e.g., a difficult math problem). In contrast

the performance is not affected when the test is

easier and less complex. Various models have been

proposed to explain the negative effects of stress.

These models assume that stress includes activation

of considerations unrelated to the specific learning

task limiting students’ engagement to the learning

activity and thus causing greater effort. Students

who are worried about a failure can not focus on

learning process. Stress decreases students' interest

and internal motivation. But in some cases stress can

motivate students to invest extra effort to avoid

failure.

It would not be correct to assume that positive

emotions always cause positive results and negative

emotions cause negative results. Instead, the results

depend on the mediation procedures and specific

requirements of the project under consideration.

More specifically, positive emotions, such as

activating the pleasure of learning are probably

beneficial for the students' performance in most

circumstances. Negative emotions such as the

deactivation, the despair and boredom can be

assumed as inconvenience to any kind of academic

performance. Also, the effects of positive emotions

of deactivation like relief and relaxation, often is

harmful. Similarly, negative emotions such as stress,

shame, and anger may exert ambiguous

effects,

reducing the attention and interest, or enhancing the

student's external incentives for greater effort and

better performance. Therefore, trigger of negative

emotions can improve performance in specific cases,

although in most cases affects negatively.

2.2 Adaptivity in Students’ Emotions

The emotional regulation includes the augmentation

of the enjoyment of learning and the reduction of the

anxiety. Emotional intelligence has developed

cognitive skills for this purpose (Matthews, et al.

2002). In the academic context, the treatment

focused on the problem seems to be the most

appropriate adjustment (e.g. with intensive effort to

better prepare to perform to a test). The emotion-

oriented treatment may also be an adaptive solution.

Dealing with this treatment includes relaxation

techniques, avoiding stressful thoughts and

biofeedback techniques. The following factors

which are under teachers’ control seems to be

important to students’ sensation.

 Teaching quality. Factors, such as lack of

structure, lack of clarity and excessive demands are

known to enhance the students' anxiety during the

test (Zeidner, 1998). Instead, well-structured

teaching and clear explanations may increase the

students’ capacity to control their feelings.

 Quality of teaching motivation. There are two

important ways to motivate students based on

academic values and cause various emotions. First,

if the learning environment meets the students’

needs, then it is likely to create positive emotions.

Second, the teachers’ enthusiasm can facilitate the

USING SENSORS TO DETECT STUDENT’S EMOTION IN ADAPTIVE LEARNING ENVIRONMENT

adoption of positive students’ feelings through

observation and emotional transmission (Hatfield, et

al. 1994).

 Support of autonomy and self-regulated learning

in order to achieve emotional control.

 Creating appropriate structures that learning

goals are achieved. There are learning practices

involving individualism, competition and

collaboration structures in the classroom (Johnson &

Johnson, 1974). The learning structures promote

students’ successful efforts to maintain control over

their emotions.

3 AFFECTIVE COMPUTING

The work on synthesis and analysis of emotions is

an interdisciplinary scientific field consisting from

the combination of computer science, psychology

and cognitive science (Allen and Carifio, 1995).

3.1 Galvanic Skin Response (GSR)

This technique is associated with the change of

electrical properties of the skin when external

voltage is applied. This is a test of the sweat

function, which measures the change in conductivity

of the skin during the flow of low voltage current

after a stimulus. The recording of the conductivity

(or the inverse of conductivity i.e. resistance) is

based on the application of external constant voltage

to the skin (Lykken & Venables, 1971). Then the

voltage across a fixed resistor in parallel with the

resistance of the skin is measured. This work will

attempt to proceed deeper in the design and the

operation of a GSR sensor. The edges of the human

body (hands and feet) have a very high proportion of

sensory nerves endings and so they become ideals

for the application of skin resistance measurements.

3.2 GSR Sensor Design

The electrical circuit is connected to an Arduino

duemillanove (

http://www.arduino.cc/) which is used

as an analog to digital converter. Two leads are

attached to two fingertips. One lead sends current up

to 5V and it is connected to the power pin of

arduino. The other is split into two wires the one is

connected to the analog pin 0 of arduino and the

other wire is connected to a 110 KOhm resistor and

then to the ground.

Figure 1: The GSR sensor electrical circuit.

The open-source electronics prototyping

platform Arduino duemillanove is programmed in

such a away to send the bio-signal value through the

USB port to a computer application. This setup

measures, GSR bio-signals every 30 milliseconds.

The values are read from arduino analog(0) and

they imply the change in resistance of the voltage

going through the body. Every student has an

identical user code. Every value read is graphed, and

a progressive average is calculated to smooth out the

values. A baseline reading is taken for 10 seconds if

the readings go flat (fingers removed from leads).

The progressive average value of measurements for

each student is inserted in a database table with the

student’s code and the timestamp. The measurement

range is from 0 to 255. The low level values have

white colour in the graph, the medium level values

have green colour, the higher values have orange

colour and the highest values have red colour.

During the learning activity a small window is

displayed to the student and he/she can easily

recognize his /her emotional state. Also the teacher

role has monitoring options and he can open display

windows for each measured student and follow up

student’s emotional state during the activity. There

are also options to reconstruct the emotional graph

of one or more students for a particular date.

Figure 2: The GSR graph for user “aa”.

INNOV 2011 - Second International Conference on Innovative Developments in ICT

3.3 Measurement Ranges

Initially for every user predefined quartiles are used

as a measure of emotions variability. The

interquartile ranges are defined as in descriptive

statistics. So every captured GSR value is classified

into four categories according to the following:

 For values between 0 and 0.25 of measurement

range, low value category with white colour. The

measured values near 0 cause low white graph lines

and may indicate boredom.

 For values between 0.25 and 0.50 of

measurement range, normal value category with

green colour indicating calm.

 For values between 0.50 and 0.75 of

measurement range, high value category with orange

colour, indicating a short of anxiety.

 For values greater than 0.75 of measurement

range, high value category with red colour,

indicating stress.

Assuming that every measured subject may have its

own sentimental quartiles, it was decided a test

measurement to precede the main activity. During

this trial activity every user is measured while

observing critical words rolling across his

application window one at a time. According to

psychologists there ore some words which nearly all

people will react to. The test measurement average

values are stored in the database in a specific table.

After the trial activity a k-means clustering

algorithm is applied on each user’s measurement,

defining the new inter-quartile range values which

are specific for that user. These new range values are

stored in the database for each user code and they

are used in the next measurements.

4 RESEARCH ISSUES

This research is considering two basic issues:

 Whether the GSR measures will operate as

scaffold to students, through biofeedback, to

regulate their emotions mainly their stress during the

learning activity and under pressure.

 Whether the GSR measures will support the

teacher to adapt the learning activity according to

students’ emotions (i.e. make the activity more

interesting if the student seems to feel boredom or

support the student to overcome a stressful

condition).

5 RELEVANT WORKS

GSR emotional detection is one of the first methods

applied from scientists mainly psychologists to

detect stress (Epstein & Fenz, 1967) and as lie

detector (Prokasy & Raskin, 1973). Now days there

are many GSR sensor designs using NXT lego,

arduino and customized circuits (i.e. Cornell

University). These all techniques were applied to

individuals while they were watching a film, hearing

some critical words, during examination or watching

a lecture. They did not so far consider their subjects

as members of a learning group and they did not

deal with subject’s emotions in relation to the

learning context. This work attempts to apply GSR

emotion detection as support to students and to the

teacher during a collaborative learning activity.

6 THE COLLABORATIVE

ACTIVITY SUPPORTED BY

THE GSR MEASUREMENT

The student collaboration was supported by a

videoconference tool called Big Blue Button

(

http://bigbluebutton.org/). The activity was separated

into two phases. Two collaborative techniques were

used jigsaw (Gallardo, et. al. 2003) and fishbowl

(Leonard, et al. 1999). Jigsaw was adopted because

it is an effective learning technique with increasing

positive educational outcomes. The jigsaw process

encourages listening, engagement, and empathy by

giving each member of the group an essential part to

play in the academic activity. The fishbowl

technique was used because it allows an entire group

to participate in a conversation. During the first face

there were two expert groups composed of six and

seven students each. Then at the second face there

were four groups with two to four students each. The

GSR measure was applied to a group of four

students. The support from the sensory emotion

estimate was impressive. The student that initialises

the presentation of the group assignment was very

anxious and her GSR values were continuing to be

very high (red range). The teacher reacting to this

condition asked from another member of this group

to continue the presentation. At first the new

representative was very anxious too, but later on he

regulated his stress and finally his graph was green.

The teacher noticed that in the mean time the

measures of the other two students of the group that

were silent were very low (white range). So, he

asked them questions about their participation into

USING SENSORS TO DETECT STUDENT’S EMOTION IN ADAPTIVE LEARNING ENVIRONMENT

the group project and he assigned to them a more

active role. There was a significant reaction by the

one student displayed on her monitoring window,

showing some anxiety but not stress. The other

student continued to be calm. After this activity the

students that used the GSR sensor filled a

questionnaire.

6.1 Evaluation of the GSR Support

The research process followed the interpretative

approach. The structure of the questionnaire

followed the Unified Theory of Acceptance and Use

of Technology (UTAUT) (Venkatesh et al.,

2003).The applied UTAUT model was based into

four core determinants performance expectancy,

effort expectancy, social influence, facilitating

conditions; and three control variables: gender,

experience, and voluntariness of use.

Figure 3: The applied framework.

The survey subjects were one male and three

females postgraduate Master’s degree students.

Since the sample was too small, there are not

expectations for safe conclusions. Nevertheless it is

a starting research and there is the opportunity to

have an initial feeling about it. After completing the

questionnaire the students were interviewed

separately. All of the students found the GSR

application very helpful and easy in order to realise

their feelings. The three students, one male and two

females considered that the GSR graph supported

them in dominating their anxiety through

biofeedback and they think that this self-regulation

resulted in better performance. The male student

regulated his stress in longer time than the two

females. The fourth student (female) failed to

interact with the GSR application through

biofeedback due to technical problems. All students

suggested that this is a useful support to learning

activities and they would be willing to reuse this

GSR sensor in other academic activities.

7 CONCLUSIONS

This work is a case study of an initial effort to detect

student’s emotions mainly stress and boredom

during learning activities. Furthermore it tries to

apply adaptive methods in order to succeed emotion

regulation. Considering this scope, the whole

activity was successful and the initial feeling was

that student emotion sensation is a very useful

support for the students themselves and at the same

time for the teacher. Three of the students interacted

with the

GSR sensor and through biofeedback they

launched self regulation reactions. The teacher

through the GSR measures had a further support to

realize his students’ feelings and reacted adaptively.

8 FUTURE WORK

It is intended to repeat collaborative learning

activities with GSR measurements applied to more

student groups. It must be reported that there was an

initial skeptical attitude towards the use of the GSR

sensor by many of them. But after the described

activity this attitude has been weakened. Also it is

intended to add more emotion detection techniques

in order to have more accurate emotion sensations.

The emotion monitoring will be embedded into the

collaborative tool being accessed by significant

permissions. A new role will be introduced as

emotional moderator assigned to an appropriate

member of each group. An embedded to the

collaborative tool intelligent agent will give support

to each student in order to regulate his emotional

state and to the group moderator to adaptively react.

The teacher will monitor through a real time

protocol the emotional state of his students.

REFERENCES

Allen, B., Carifio, J., 1995. Methodology for the analysis

of emotion experiences during mathematical problem

solving. Annual Conference of the New England

Educational Research Organization. Portsmouth.

Csikszentmihalyi, M., 1990. Flow: The Psychology of

Optimal Experience. Harper-Row, New York.

Dweck, C., 2002. Messages that motivate: How praise

molds students’ beliefs, motivation, and performance

(in surprising ways). Improving Academic

Achievement: Impact of Psychological factors on

Education pp. 61-87. Academic Press, Orlando.

Epstein, S., Fenz.W., 1967. The detection of areas of stress

through variations in perceptual threshold and

INNOV 2011 - Second International Conference on Innovative Developments in ICT

physiological arousal. Journal of Experimental

Research in Personality, 2,191-199.

Festinger, L., 1957. A theory of cognitive dissonance.

Stanford University Press, Stanford.

Fielder, K., 2001. Affective states trigger processes of

assimilation and accommodation. In: Martin, L. &

Clore, G. (eds.) Theories of Mood and Cognition: A

User’s Guidebook. pp. 85-98. Erlbaum, Mahwah.

Gallardo T., Guerrero L. A., Collazos C., José A. P.,

Ochoa S., 2003. Supporting JIGSAW-type

Collaborative Learning. Department of Computer

Science, Universidad de Chile, Blanco Encalada.

Graesser, A., Lu, S., Olde, B., Cooper-Pye, E., & Whitten,

S., 2005. Question asking and eye tracking during

cognitive disequilibrium: Comprehending illustrated

texts on devices when the devices break down.

Memory and Cognition 33, 1235—1247.

Graesser, A., Olde, B., 2003. How does one know whether

a person understands a device? The quality of the

questions the person asks when the device breaks

down. J. of Educational Psychology 95, 524—536.

Hatfield, E., Cacioppo, J. T., & Rapson, R. L., 1994.

Emotional contagion. New York: Cambridge

University Press.

Isen, A., Daubman, K., & Nowicki, G., 1987. Positive

affect facilitates creative problem solving. J. of

Personality and Social Psychology 52, 1122-1131.

Isen, A., 2001. An influence of positive affect on decision

making in complex situations: Theoretical issues with

practical implications. J. of Consumer Psychology 11,

75-85.

Johnson, D. W., Johnson, R. T., 1974. Instructional goal

structure: Cooperative, competitive or individualistic.

Review of Educational Research, 4, 213–240.

Kuhl, J., 1983. The functional significance of emotions in

perception, memory, problem-solving, and overt

action. Sprache & Kognition 2, 228—253.

Kumar, R., Rosé, C. P., Wang, Y. C., Joshi, M., Robinson,

A. 2007. Tutorial dialogue as adaptive collaborative

learning support. In R. Luckin, K. R. Koedinger, & J.

Greer (Eds.) Proceedings of Artificial Intelligence in

Education (pp. 383-390). IOS Press.

Lazarus, R., 1991. Emotion and adaptation. Oxford

University Press, New York.

Leonard W. J., Dufresne R.J., Gerace W. J., Mestre J. P.,

1999. Collaborative Group Techniques. A discussion

of teaching via small-group cooperative learning work.

Lewis, M., Haviland-Jones, J. M., 2000. Handbook of

emotions. New York: Guilford Press.

Lykken, D. T., Venables, P., 1971. Direct measurement of

skin conductance: A proposal for standardization.

Psychophysiology, 8, 656–672.Designated a Citation

Classic, Institute for Scientific Information.

Mandler, G., 1984a. Another theory of emotion claims too

much and specifies too little. Current Psychology of

Cognition 4, 84-87.

Mandler, G., 1984b. Mind and body. W. W. Norton, New

York.

Olafson, K. M., Ferraro, F. R., 2001. Effects of emotional

state on lexical decision performance. Brain and

Cognition, 45, 15–20.

Piaget, J., 1952. The origins of intelligence in children.

International University Press, Oxford.

Prokasy, W. F. & Raskin, D. C., 1973. Eds.

Electrodermal Activity in Psychological Research.

New York: Academic Press.

Rummel, N., Weinberger, A. 2008. New challenges in

CSCL: Towards adaptive script support. In G.

Kanselaar, V. Jonker, P.A. Kirschner, & F. Prins,

(Eds.), International perspectives of the learning

sciences: Cre8ing a learning world. Proceedings of the

Eighth International Conference of theLearning

Sciences (ICLS 2008), 3 (pp. 338-345). International

Society of the Learning Sciences, Inc.

Schutz, P. A., Pekrun, R., 2007. Emotion in education.

San Diego, CA: Academic Press.

Schwarz, N., Skurnik, I., 2003. Feeling and thinking:

Implications for problem solving. In: Davidson, J. &

Sternberg, R. (eds.) The Psychology of Problem

Solving. pp. 263-290. Cambridge University Press,

New York.

Spering, M., Wagener, D., & Funke, J., 2005. The role of

emotions in complex problem–solving. Cognition and

Emotion 19, 1252—1261.

Stein, N.L., Hernandez, M.W., 2007. Assessing Emotional

Understanding in Narrative On- line Interviews: The

Use of the Narcoder. In Coan, James A. and Allen,

John J. B. (Eds.), Handbook of Emotion Elicitation

and Assessment. Oxford University Press: New York.

Zeidner, M., 1998. Test anxiety: The state of the art. New

York: Plenum Press.

Zeidner, M., 2007. Test anxiety in educational contexts:

What I have learned so far. In P. A. Schutz & R.

Pekrun (Eds.), Emotion in education (pp. 165–184).

San Diego, CA: Academic Press.

USING SENSORS TO DETECT STUDENT’S EMOTION IN ADAPTIVE LEARNING ENVIRONMENT