Measuring Student Emotions in an Online Learning Environment

Márcio Alencar and José Francisco Netto

Institute of Computing, Federal University of Amazonas, Manaus, Brazil

Keywords: Sentiment Analysis, Affective Computing, Multi-Agent System, Learning Management System, Framework.

Abstract: The use of Virtual Learning Environments by educational institutions grows every year. In these

environments, students use asynchronous and synchronous tools to communicate, express their opinions on

various subjects. All of this information can be used to identify students' emotional states and improve

education. Using sentiment analysis techniques is possible to identify students with difficulty, frustration,

discouragement, this approach helps detect students with potential dropout. In this paper, an experiment was

carried out with students of a technical course using a tool that identifies the emotional state of a class and

their students using sentiment analysis on student's posts forum. The results demonstrate that the approach

proposed in this paper can support teachers in monitoring students' emotional states by accessing and

analyzing discourse forums to assist in decision making and learning improvements.

1 INTRODUCTION

In recent years, Artificial Intelligence (AI)

researchers have been trying to equip systems to

interpret emotions and Sentiment. Emotions and

Sentiment play a key role in our daily lives as they

help in decision making, learning, and

communication (Poria et al, 2017).

Virtual Learning Environment has several tools

used by teachers and students, among which we can

highlight the discussion forums, which are

communication tools that allow for collaboration

between those involved, where students can show

emotions (Cercel et al, 2015).

In the classroom the teacher can identify the

affectivity through the expressions, dialogues, and

behavior of the students, however in distance

courses this situation is more challenging, since the

students’ emotions are registered in the

communication tools such as: forum, chat, Journal,

message exchange between students and teachers,

etc (Mohammad, 2016)

In this context, the texts produced by students we

can identify many feelings, such as happiness, fear,

surprise and with can we use this information as

another tool to help teachers in their daily activities

(Alencar and Netto, 2017).

According to research conducted by (Fei and Li,

2018), affective information can help teachers to

improve their pedagogical practices. Thus,

perceiving the affectivity in the Virtual Learning

Environments, captured through its communication

tools, can be another resource to verify the needs of

the students.

The affective bond between teacher and student

plays an important role in pedagogical mediation,

since the students expect the tutor to be attentive,

motivating, and encouraging in the virtual and in-

person moments; that is, the students feel more

encouraged and secure when there is an affective

and cognitive relationship within the activities

(Roorda et al, 2017).

According to (Grawemeyer et al, 2017) students

during learning move between the positive and

negative affective state, for the teacher it is

interesting to know how to identify the students'

affective state, being able to identify which elements

are related to positive and negative aspects.

Emotions may be related to certain behaviors, for

example: negative emotions (confusion, frustration,

discouragement, anxiety and anger), while positive

emotions may be related to increased dedication,

participation, motivation and interest in the course.

The methodology adopted in this paper began

with a literature review and systematic mapping of

Multi-Agent System, Affective Computing and

Sentiment Analysis, to understand their

performance, which technologies are used in the

implementation of the experiments and their

contributions to education.

Alencar, M. and Netto, J.

Measuring Student Emotions in an Online Learning Environment.

DOI: 10.5220/0008956505630569

In Proceedings of the 12th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2020) - Volume 2, pages 563-569

ISBN: 978-989-758-395-7; ISSN: 2184-433X

563

Based on these studies, this paper conducted an

experiment with students of the technical

specialization course to identify the emotional state

of students through your e-learning posts. To help

the teacher this was developed a system integrated

with the Virtual Learning Environment, which uses

a multi-agent system to collect messages posted by

students in discussion forums and through

techniques of sentiment analysis to identify the

emotional state of students.

Sentiment Analysis can reveal useful information

about students and can help understand student

behavior and improve learning. Through the

messages posted in the forums, each student

expresses his opinion individually, part of our study

is to identify the emotional state of students to

collaborate with teachers (Mohammad, 2016).

One of the challenges faced by those working

with education is to motivate students to learn. The

motivation of this work is to demonstrate that from

an approach using sentiment analysis it is possible to

know the affective aspects of the students, favoring

pedagogical actions, being able to assist teachers in

decision making and intervention when necessary,

such as identifying discouraged students,

unmotivated, who wish to give up the course and

avoid school drop-out (Gontzis et al, 2017).

The rest of this paper is structured as follows.

Section 2 presents related works. The experiments

are described in section 3, section 4 presents the

results and discussions about the experiments. The

paper is finished with section 5, which are the

conclusions and future works.

2 RELATED WORK

This research began with a bibliographic review and

systematic mapping about Multi-Agent System,

Affective Computing and Sentiment Analysis, to

identify the main works, concepts and technologies

used in the literature that helped in the development

of this research.

2.1 Multi-Agent System

The Multi-Agent Systems (MAS) are a type of

distributed Artificial Intelligence system composed

of agents that act in an environment who interact

with each other, seeking to solve problems in a

collaborative way. Its main characteristics are:

Social Organization, Cooperation, Coordination,

Control and Communication (Deloach, 2001).

According to our literature review we can see a

growth in research using MAS technology in

education, among them we can name the work

(Lima et al, 2018) carried out a Systematic Mapping

of the Literature on Intelligent Agents and Multi-

Agent Systems in the educational context. The

researchers selected 84 papers using MAS applied in

the area of Educational Informatics and from these

articles were identified 20 papers using MAS to

assist distance learning and Virtual Learning

Environment.

Many students express their emotions in the

online forums and assessing this vast amount of

information requires many hours of teacher work, so

researchers (Alencar and Netto, 2017) have

developed an Virtual Agent integrated into an

Moodle Virtual Learning Environment that uses a

multi-agent system to monitor student activities and

identify emotional state, thereby collaborating with

the teacher in teaching and learning.

Taking into account that every year the number

of students in distance courses grows, so it is good

work for teachers to accompany all these students,

so it is important to use intelligent agents. An Multi-

Agent System can perform various tasks in e-

learning, such as monitoring user activity, capturing

information automatically, and performing custom

recommendation of educational content. Despite

their wide applicability, there are still a number of

challenges faced by Multi-Agent System including

coordination between agents, security, and task

allocation (Dorri et al, 2018).

The researchers (Fontes et al, 2017) have

observed that student tracking is often the same

irrespective of their performance and behavioral

differences in the environment, thereby creating an

intelligent agent-based learning environment model

inspired by intelligent mentoring to provide

adaptability to Moodle Virtual Learning

Environment, taking into account the performance of

students in tasks and activities proposed by the

teacher.

2.2 Affective Computing

Affective Computing is the area of computer science

that seeks to recognize and represent affectivity in

human-machine interaction, that is, the use of

emotions in different aspects in computer systems

(Poria et al, 2017).

In the educational field we can use Affective

Computing to extract aspects related to affectivity,

such as emotions and personality, in order to offer

the student a more affective learning environment.

ICAART 2020 - 12th International Conference on Agents and Artiﬁcial Intelligence

564

Through Artificial Intelligence techniques, the

machine can identify the student's affective state,

understand and act, using computational algorithms

to evaluate and respond to affective states (Picard,

2003).

In everyday life we come across a number of

different situations and develop our own feelings for

each of them. These feelings generate emotions that

can be recognized through some body expressions,

however in Virtual Learning Environments this

situation is different because it is necessary social

interaction of students through the activities and

writing, so they can get involved and share ideas

demonstrating their affective relationships. For (Liu,

2015) the feeling represents an attitude, opinion or

emotion that the opinion maker has about the target

of the same.

For (Kumi-Yeboah et al, 2017) the development

of the individual begins with the interactions that the

individual establishes in the historical and cultural

context in which he is inserted. The construction of

knowledge occurs from an intense process of social

interaction, where we can highlight the language,

which has two fundamental characteristics that are

communication and the construction of thought.

According to (Zheng et al, 2015), students are

encouraged during the teaching-learning process to

use various interaction mechanisms that allow the

generation of texts. In addition to situations of

objective nature, where textual messages related to

the answers to questions or indication of results

obtained are observed, situations are observed in

which texts describing subjectivities are generated.

These may correspond to a variety of situations,

ranging from student comments on their

performance, personal observations on peer and

teacher comments, indications of the degree of

satisfaction and acceptance with the teaching

processes, or the description of difficulties

encountered by students in participate in this type of

education.

In distance learning courses the mediation of the

educational process occurs through a virtual learning

environment, which needs synchronous and

asynchronous communication tools, such as: forum,

chat, diary, message exchange, etc.). These

resources help the cognitive and affective

development of students, both individually and

collectively (Fei and Li, 2018).

2.3 Sentiment Analysis

Sentiment Analysis is an area of study within

Natural Language Processing that is concerned with

identifying the mood or opinion of subjective

elements within a text (Zhang and Liu, 2017). Use

techniques from various fields of computing such as

natural language processing, information retrieval,

data mining, and statistics.

Its application is very wide, people can express

their opinion about products, services, brands,

health, politics, education, and even in the music

field, how can we highlight the work of (Madhok et

al, 2018).

Millions of people around the world use social

networks, so the volume of data grows every

moment, making it a huge challenge to manipulate

all this information and extract useful information

that can contribute to businesses (Rodrigues et al,

2016).

A person expressing an opinion expresses a

feeling that represents attitude, opinion or emotion.

Feeling can be measured by its polarity (positive,

negative or neutral), but it can also be measured by

emotion classes (Liu, 2015).

Sentiment Analysis is applied in several surveys,

to get an idea of the size of this area, we can observe

the work of (Pang and Lee, 2008), called "Opinion

Mining and Sentiment Analysis", has been cited by

more 9.000 paper to date.

The work of (Piryani et al, 2016) carried out a

systematic mapping to analyze papers published in

the period from 2000 to 2015, on Opinion Mining

and Sentiment Analysis (OMSA). In a more detailed

analysis they observed that the approaches (machine

learning and lexicon based) are more popular in

publications.

Considering the large amount of data on the

Internet, we find in the literature that many

researchers create framework using Sentiment

Analysis to assist in this task (Dragoni et al, 2016).

SeNTU framework (Chikersal et al. 2015) is a

project that uses sentiment analysis proposed by

students and staff members of NTU (Nanyang

Technological University). The main objective of

the framework is to prove that the use of different

paradigms, for example machine learning, linguistics

and knowledge representation, can improve the

performance of sentiment analysis. The SeNTU

framework does sentiment analysis using two

classifiers: rule-based and supervised. In the

preprocessing phase, text is normalized and

tokenized, then a combination of lexical classifiers,

such as SenticNet, is used to infer the polarity of

sentences, same tool in the research presented in this

paper.

Measuring Student Emotions in an Online Learning Environment

565

3 EXPERIMENT

The School of Distance Professional Education

(CETAM EaD) works with distance learning courses

using Moodle (http://ead.cetam.am.gov.br/) and

serves various municipalities in the state of

Amazonas, Brazil. To evaluate the feasibility of the

proposal and gain mastery of technical tools, we

conducted an experiment that used real data from

students of the technical specialization course in

Information Technology Management, in the

discipline "Computer Network Management", which

used two activities (Forum and Assignment)

highlighted in Figure 1 (in Portuguese).

Figure 1: Course “Computer Network Management”.

This experiment used real data from four (4)

classes of this course, completed in May 2018, with

an average of 40 students each.

To perform this experiment an architecture was

developed composed of a Multi-Agent System

integrated to the Virtual Learning Environment in

charge of collecting the messages posted by the

students in the discussion forums, then processing

these messages using sentiment analysis and finally

the agents present the results in graphical form. The

MAS has 3 agents, their function is described in

table 1.

Table 1: Function of the agents.

Agent

Function

Collector

Collect messages posted by students

Sentiment

Extract the emotion/polarity of messages

Presenter

Presents the result of the Sentiment Analysis

In the 80's, Robert Plutchik created the Wheel of

Emotions (Plutchik, 1984), which was a reference

for our research, an eight-pointed star, in which each

of these represents a primary feeling with opposing

pairs, representing a total of 48 emotions, as we can

see in Figure 2.

Figure 2: Wheel of Emotions (Plutchik, 1984).

Researchers (Cambria et al, 2018) used in their

research a new category of emotion categorization,

called Hourglass of Emotions, based on the

emotions wheel proposed by (Plutchik, 1984). The

Hourglass Model organizes the primary emotions

around four affective dimensions (Pleasantness,

Attention, Sensitivity, Aptitude). Each affective

dimension is characterized by six septic levels.

These levels have a set of 24 basic emotions, six

emotions for each affective dimension, which are

used in the association of emotion with text, as we

can see in Figure 3.

Figure 3: Four dimensions of the hourglass model

(Cambria et al, 2018).

In this project the multi-agent system uses the

SenticNet tool to perform the sentiment analysis of

the texts produced by the students. According to

(Cambria et al, 2018) SenticNet is a set of tools that

performs opinion mining, sentiment analysis and

explores Artificial Intelligence and Semantic Web

techniques, using a knowledge base available in 40

languages. The SenticNet knowledge base has

undergone several updates over the years, in version

1 it had 6.000 concepts, in version 2, 13.000

concepts, in version 3 30.000 concepts, in version 4,

50.000 concepts with the accuracy of 91.3% and in

the current version 5 has 100.000 concepts with the

accuracy of 94.6%.

ICAART 2020 - 12th International Conference on Agents and Artiﬁcial Intelligence

566

The MAS used in this experiment is the same as

applied in the work of (Alencar and Netto, 2017),

these agents perform their activities following three

steps: data collection, classification and summary.

In the data collection phase, MAS collected 2016

messages posted in the forums by students of this

course (Table 2). After this each sentence has been

handled, agents remove the HTML markers, then

each message is translated into English using Google

Translator, stopwords are removed, and finally the

message is divided into words.

Table 2: Messages posted in forum.

Class

Messages quantity

487

528

655

346

Total

2016

In the classification phase, MAS uses the

SenticNet 5 tool, capable of able to identify polarity

(-1 to 1) and emotion (admiration, joy, interest,

anger, disgust, sadness, surprise and fear) portrayed

in texts published by students.

At the summary stage, the SMA identifies the

emotions of the students in a class regarding the

texts posted on the forum, as well as the polarity

value, and represents these values in the form of a

pie chart.

4 PARCIAL RESULT

In this section, we will present the results of the

sentiment analysis about the texts posted by the

classes. In figure 4 we present 4 graphs, each graph

represents the emotional state of each class (Class 1,

Class 2, Class 3, Class 4).

We can see in each graph that there are 8

emotions (admiration, joy, interest, anger, disgust,

sadness, surprise and fear) classified using the

SenticNet tool, each with a different color and equal

the colors of the wheel of the emotions.

In graph 3 we have class 3, which presents some

negative feelings from students that can help in

decision making, have more students with a negative

emotional state, which is 17,32 % disgusted, each of

these emotions is represented by a different color,

larger values on the graph need to be analyzed as

they can represent positive or negative emotions.

In the results presented in the Graphs, we can

observe the importance of the Sentiment Analysis in

Figure 4: Emotional state of the classes.

the texts produced by the students in each class.

With this information we can observe the emotional

state of the students during the course, for example

knowing which class is having the most difficulty

understanding a subject, so the teacher can measure

the mood of the students in real time, helping to

make decisions during the course, avoiding dropping

out, dropping out, etc. According to (Mohammad,

2016) this information can help the teachers to check

the difficulties of the students in carrying out each

activity, so we can collaborate teachers in their daily

activities.

Monitoring students during an online course is a

time-consuming activity for teachers, hence the

importance of having tools that can understand what

is happening in the virtual environment and can

inform the teacher (Alencar and Netto, 2011).

As a limitation of the project we can highlight

the difficulty in handling a large volume of data and

the difficulty in identifying emotion in texts.

5 CONCLUSION

In this work we present an experiment using real

data from students of distance courses, through the

analysis of feelings we can identify which feeling

predominates most in a class. We verified that

through the approach we have the perception of the

feelings of the students, with this we can identify

problems and anticipate actions carried out by the

teachers.

The use of Sentiment Analysis Techniques in

educational systems helps to identify the emotion

expressed by students, understanding emotions can

Measuring Student Emotions in an Online Learning Environment

567

both positively and negatively, enabling yet another

support tool for teachers.

The experiment demonstrated an efficiency in

using uses Multi-Agent System as technology for the

proposed approach, considering the proactivity and

communication of the agents, another highlight is

the use of graphics with the emotional state that

facilitates interpretation by teachers.

As a future work, we intend to develop a system

integrated with the Virtual Environment that

conducts Sentiment Analysis in real time, presents

different graphs and checks the emotional state of

each student and each class.

REFERENCES

Alencar, M.A.S.; Netto, J.F.M. (2011) Improving

Cooperation in Virtual Learning Environments Using

Multi- Agent Systems and AIML. Frontiers in

Education Conference - FIE.

Alencar, M.A.S.; Netto, J.F.M. (2017) Melhorando a

Colaboração de um Ambiente Virtual de

Aprendizagem usando um Agente Pedagógico

Animado 3D. XXVIII Brazilian Symposium on

Informatics in Education, v. 1, p. 1417, 2017.

Cambria, E.; Poria, S.; Hazarika, D.; Kwok, K. SenticNet

5: Discovering Conceptual Primitives for Sentiment

Analysis by Means of Context Embeddings. In: AAAI,

pp. 1795-1802 (2018).

Cercel, D.;Trausan-Matu1, S. Modeling Post-Level

Sentiment Evolution in Online Forum Threads. In:

Proceedings of the 7th International ICAART

Conference on Agents and Artificial Intelligence,

Lisbon, Portugal (2015).

Chikersal P.; Poria, S.; Cambria, E. (2015) SeNTU:

Sentiment Analysis of Tweets by Combining a Rule-

Based Classifier with Supervised Learning. In:

Proceedings of the international workshop on semantic

evaluation. Denver, Colorado, USA, SemEval 2015.

Dorri, A; Kanhere, S. S. ; Jurdak, R. Multi-Agent

Systems: A survey. IEEE Access, vol. 6, pp. 28573–

28593, Jul. 2018.

Dragoni, M; Da, C.; Pereira, C.; Tettamanzi, A.G.; Villata,

S. Smack: An argumentation framework for opinion

mining, Proceedings of the Twenty- Fifth International

Joint Conference on Artificial Intelligence, pp.9-15,

2016.

Deloach, S. A.; Wood, M. Developing Multiagent Systems

with agentTool. In: Proceedings of Lecture Notes in

Artificial Intelligence. Springer – Verlag. Berlin, 2001.

Fei, H., Li, H. (2018). The Study of Learners’ Emotional

Analysis Based on MOOC. In: Xiao J., Mao

ZH.,Suzumura T., Zhang L. J. (eds) Cognitive

Computing – ICCC 2018. ICCC 2018. Lecture Notes

in Computer Science, vol 10971. Springer, Cham

Fontes, L. M. O.; Valentim, R. A. M.; Neto, F. M. M.;

Souza, R. C. A. Multi-Agent Architecture for

Monitoring Tutoring Activities. In VLEs. IEEE Latin

America Transactions, vol. 14, no. 10, pp. 4327-4333,

2016.

Gontzis, A.F.; Karachristos, C.V.; Panagiotakopoulos,

C.T.; Stavropoulos, E.C.; Verykios, V.S. Sentiment

Analysis to track Emotion and Polarity in Student

Fora, Proc. of PCI 2017.

Grawemeyer, B.; Mavrikis, M.; Holmes, W.; Gutiérrez-

Santos, S.; Wiedmann, M.; Rummel, N. (2017).

Affective Learning: Improving Engagement and

Enhancing Learning with Affect-Aware

Feedback. User Modeling and User-Adapted

Interaction, 27(1), 119–158.

Guo, S.; H¨ohn, S.; Xu, F.; Schommer, C.: PERSEUS: A

Personalization Framework for Sentiment

Categorization With Recurrent Neural Network. In:

International Conference on Agents and Artificial

Intelligence, Funchal 16-18 January 2018. p. 9 (2018)

Kumi-Yeboah, A.; Dogbey, J.; Yuan, G. (2017). Online

Collaborative Learning Activities: The Perceptions of

Culturally Diverse Graduate Students. Online

Learning, 21(4), 5-28. doi: 10.24059/olj.v21i4.1277

Lima, D. P. R.; Gerosa, M. A.; Netto, J. F. M. Using

Awareness Information to Enhance Online Discussion

Forums: A Systematic Mapping Study. In: Frontiers in

Education, 2018, San Jose, CA. 2018 Frontiers in

Education Conference - Fostering Innovation through

Diversity, 2018.

Liu, B. Opinions, Sentiment, and Emotion in Text.

Cambridge University Press, p. 381, 2015.

Madhok, R; Goel, S; Garg, S. SentiMozart: Music

Generation based on Emotions. In: Proceedings of the

10th International ICAART Conference on Agents and

Artificial Intelligence, Madeira, Portugal (2018).

Mohammad, S. M. (2016). Sentiment Analysis: Detecting

Valence, Emotions, and Other Affectual States from

text. Emotion Measurement

Pang, Bo; Lee, L. (2008), Opinion Mining and Sentiment

Analysis, Foundations and Trends in Information

Retrieval: Vol. 2: No. 1–2, pp 1-135

Picard, R.W. (2003). Affective Computing: Challenges.

International Journal of Human-Computer Studies,

Volume 59, Issues 1-2, July 2003, pp. 55-64.

Piryani, R. ; Madhavi, D; Singh, V. K. Analytical

Mapping of Opinion Mining and Sentiment Analysis

Research During 2000-2015, Inf. Process. Manag.,

vol. 53, no. 1, pp. 122-150, 2016.

Plutchik, R. (1984). Emotions: A General

Psychoevolutionary Theory. Approaches to Emotion,

1984, 197-219.

Poria, S.; Cambria, E.; Bajpai , R.; Hussain, A. 2017. A

Review of Affective Computing: From Unimodal

Analysis to Multimodal Fusion. Information Fusion.

Rodrigues, R.G.; das Dores, R.M.; Camilo-Junior, C.G.;

Rosa, T.C. SentiHealth-Cancer: a Sentiment Analysis

Tool to Help Detecting mood of patients in online

social networks. Int. J. Med. Inform. 85, 80–95 (2016)

Roorda, D. L., Jak, S., Zee, M., Oort, F. J., & Koomen, H.

M. Y. (2017). Affective Teacher–Student

Relationships and Students’ Engagement and

ICAART 2020 - 12th International Conference on Agents and Artiﬁcial Intelligence

568

achievement: a meta-analytic update and test of the

mediating role of engagement. School Psychology

Review, 46, 239–261. doi:10.17105/spr2017-

0035.v46-3

Zhang L.; Liu B. (2017) Sentiment Analysis and Opinion

Mining. In: Sammut C., Webb G.I. (eds) Encyclopedia

of Machine Learning and Data Mining. Springer,

Boston, MA.

Zheng, S., Rosson, M. B., Shih, P. C., & Carroll, J. M.

(2015). Understanding Student Motivation, Behaviors

and Perceptions in MOOCs. Proceedings of the 18th

ACM Conference on Computer Supported

Cooperative Work & Social Computing - CSCW ’15.

Measuring Student Emotions in an Online Learning Environment

569