Towards using Multimodal Features of Social Networks for Improved
Contextual Emotion Detection
Ahmed S. Rizk, Sherif G. Aly and Mohamed Shalan
Computer Science and Engineering Department, The American University in Cairo (AUC), Cairo, Egypt
Keywords: Social Networks, Mood, Emotion, Pervasive, Multimodal.
Abstract: Social networks are valuable source of information that could be used in classifying users’ emotions. In this
paper, we explore the importance of certain multimodal features of social networks, other than text, that can
be used in enhancing emotion detection. We study the types of posts, the degree of interaction with contacts,
and the influence of contact opinions and how they tend to affect the emotions of social network users. We
conducted an online survey targeting Facebook users to know how they are affected by such features. The
results of our study show that status messages are the most used feature to express the social network users’
emotions, and the emotions of social network user are affected by posts and updates from friends, especially
close friends. The number of likes expressed to social network users was found to positively affect their
emotions. We will use such findings to prototype a system for enhanced emotion detection.
1 INTRODUCTION
Social networks have become an extremely valuable
goldmine of context information that can be used
very effectively in pervasive systems. One of the
numerous pieces of high-level context information
that can be elicited from social networks is human
emotions. Unfortunately, much of the relevant
scientific literature dealt with emotion detection
from social networks as a typical text mining
problem (Yassine and Hajj, 2010). However, the
multimodal features of social networks including but
not limited to the types of posts, the degree of
interaction with specific contacts, the applications
used, the influence of contact opinions and likes, are
types of channels through which enhanced emotion
detection may be achieved.
Emotion and mood are terminologies that
identify the current status of person’s cognition. An
Emotion is defined in psychology as a short-term
state of mind, which includes psychological arousal.
It cannot be a physical state like pain or a
behavioural state like aggression. For example, love,
happiness, anger, and fear are considered kinds of
emotions. Mood is also considered as a state of mind
that includes a psychological arousal however it
defers from emotion in its duration as it tends to last
longer (Walter et al., 2006). With the widespread of
social network in our daily lives, their effect on our
emotions and mood is yet to be investigated
thoroughly. We will focus on the effect of social
network on their users’ emotions in our research.
Emotions can be manifested in social networks
in situations, such as when a user is pleased for
having a nice outdoor activity, feeling down by
going through a bad day, even expressing feelings
about the world economy. Users contribute to their
social network through status updates, video posts,
likes, and comments on certain topics. Group
subscriptions can also carry much information
through which user’s emotion could be inferred.
There are many social networks on the Internet.
Facebook with close to one billion users and an
average of 130 friends per users (Facebook, 2012),
Twitter, Myspace, Google Plus, Linkedin, and Flickr
are examples of social networks.
In this paper, we realize the importance of
emotion inference as one of the important contextual
pieces of information in mobile pervasive systems.
To this effect, we study the Facebook social network
in specific and try to identify the contribution of
various multimodal features, other than text, within
the social network of a user and how those can hint
about the emotion of its users. A very insightful
review article about the future challenges and
opportunities in the domain of pervasive computing
by Conti, et al. (2012) indicates the need to
113
S. Rizk A., G. Aly S. and Shalan M..
Towards using Multimodal Features of Social Networks for Improved Contextual Emotion Detection.
DOI: 10.5220/0004305801130117
In Proceedings of the 3rd International Conference on Pervasive Embedded Computing and Communication Systems (PECCS-2013), pages 113-117
ISBN: 978-989-8565-43-3
Copyright
c
 2013 SCITEPRESS (Science and Technology Publications, Lda.)

“understand and manage the dynamics of human
behavior in order to make pervasive computing
systems more usable and tractable.” We will present
related work followed by the details of our study,
our future work and conclusion in the next sections.
2 RELATED WORK
We will analyse some of the relevant related work
that pertains to the study of emotion detection in
social systems in this section. Kramer (2012)
analysed the status updates of 400 million Facebook
users in North America over time. The author
showed that status updates provide cues to the
emotional state of the user and can provide insights
to the state of the groups updating status. He counted
the relative rates of positive and negative emotion
words used to identify culturally shared positive and
negative events. He validated that the use of positive
and negative words in status updates covaries with
self-reported satisfaction with life.
In another study, Kramer (2012) aimed to
research emotion contagion in social networks.
Emotional contagion is the process by which people
“catch” emotions form each other. He showed that
when a user exhibits a certain emotion in his or her
status, his or her friends are more likely to make
similar emotion-oriented posts.
A study was conducted by Hancock et al. (2008)
to investigate emotional communication in
computer-meditated communication. The study
examined negative emotion expression and
contagion. The authors concluded that negative
emotion was expressed and sensed by the
communicating parties and that emotional contagion
takes place in computer-meditated communication.
All these studies show that social networks are
environments where users tend to express their
emotions. However, most of them considered social
networks as a source of textual information only.
They did not take into consideration the multimodal
feature of social networks, such as likes, the degree
of interaction between users such as relationships
between users, events, gifts, and the preferences
stored in the social networks users’ profiles.
3 STUDY OF EMOTIONAL
EXPRESSION IN SOCIAL
NETWORKS
To further study how the emotions of the users of
social networks are affected by the use of social
networks, we decided to survey users in the quest for
such kind of knowledge. The aim of this survey is to
study the patterns in which the emotion of social
networks users is affected by their daily interactions.
The objective is to identify the most prominent used
features in the social network and how that can
affect emotions of the user so that we eventually can
incorporate such features in emotional detection
using social networks. The following are the
characteristics of our survey:
 Paradigm: Quantitative
 Purpose: Analytical Research
 Outcome: Applied
 Logic: Deductive Research
 Process: Quantitative
 Methodology: Cross-Sectional Surveys
In this section, we will demonstrate our research
hypotheses in details. Let the hypotheses be denoted
by the letter H. The null hypothesis is that
multimodal features of social networks have no
effect on emotions. H1: Users of social networks
express their emotions through different features of
social networks. H2: Status messages are used more
than any other feature to express emotions. H3:
When the number of likes toward one of the social
networks users increases, this positively affects the
user’s emotions. H4: Emotions of users of social
networks are affected according to the relationship
between them and the person who made the post.
(e.g. if a family member made a comment or a post
this will affect him or her emotionally more than
other posts.) H5: Receiving virtual gifts positively
affects the emotions of the social networks users.
H6: Accepting a social network event, such as
birthday, weeding …etc. will have an impact on the
emotions of the users of social networks.
3.1 Sample
A total of 220 users of social networks contributed
to this online survey. The sample consisted of
international adults of different backgrounds and
nationalities. The participants were from both
genders with age range of (18-35). We chose
Facebook as our social network as it is the most
popular of the available social networks with the
largest number of users, having close to one billion
users (Facebook, 2012). The questionnaire was
published on the Internet through an online survey
using the surveymonkey website and posted to the
researcher’s Facebook profile page; that contains
more than 410 of friends and different Facebook
pages and groups; to ensure high response rate. The
PECCS2013-InternationalConferenceonPervasiveandEmbeddedComputingandCommunicationSystems
114

total survey duration was 10 days. We used close
ended questions as the main source for this survey to
investigate the effect of the social network on its
users.
The survey questionnaire mapping matrix is
illustrated in Table 1 which shows the purpose of
question group, the number of questions related to
each group, a short description of the purpose of that
group, and the relationship between the question and
our hypotheses.
Table 1: Survey Questionnaire Mapping Matrix.
Purpose of
the question
Description
Question
number
Hypothesis
Exclusion
question
Excludes
respondents with
limited usage of
their Facebook
accounts
1 H1
Tendency to
express
emotions
through
Facebook
features
Illustrate if the
users tend to
express their
emotions through
various features
of Facebook and
being affected by
posts made by
friends
2,3 H1
Effect of likes
Explains how the
increase of the
number of likes to
a user’s post may
affect his or her
emotions
4 H3
Most used
features and
emotion
expression
Capture the most
frequently used
features by
Facebook users
and their
tendency to
express emotions
through them
5-8 H2,H5,H6
Posts that
affect the
users
emotions
Investigate, which
posts affects the
users’ emotions
the most, e.g.
posts from close
friends, family
members, work
colleagues …etc.
9,10 H4
The following section will reveal the results and
details of our survey. We will illustrate the patterns
of social network users’ behaviour.
3.2 Study Results
When asked about their daily usage of Facebook,
90% of the surveyed sample answered that they use
it on a daily basis. The graph in Figure 1 shows that
7% of the sample used Facebook at least once
weekly, 2% of the sample used Facebook at least
once monthly and only 1% does not use it. This
reflects how extensively people are keen on using
social networks and how integrated it is in their daily
lives.
Figure 1: H1: How often do online users use Facebook?
Users on Facebook post their status updates and
receive comments and likes about the posts. “Like”
is an action in Facebook where users can click Like
button that indicates their liking to the posts, the
number of likes to the posts are aggregated and
shown. Users also post photos, links, videos, and
commentary conversations, and likes are received
for those multimodal features as well. During these
interactions within the social networks, users tend to
be emotionally affected by posts, comments, and
likes made by friends and other users of social
networks. The graph in Figure 2 shows the high
tendency of users to express their emotions through
Facebook, and it also shows that friends’ posts can
affect the emotions of the social networks users.
Users of social networks read many updates from
their friends, which carry emotional implications and
these updates affect their emotions.
Figure 2: H1: Expressing emotions through Facebook.
90%
7%
2%
1%
Atleastonce
daily
Atleastonce
weekly
Atleastonce
monthly
Idonotuseit
0
10
20
30
40
50
60
70
80
Expressemotions
viafacebook
features
Emotionsare
impactedby
friends'posts
%
yes
No
TowardsusingMultimodalFeaturesofSocialNetworksforImprovedContextualEmotionDetection
115

The survey reflected that status updates, comments,
and likes are the most used features by the social
networks sample. After which users tend to use
private messaging, photos, events, and notes
prospectively. Users of social networks use status
updates, comments, and by liking their friends posts
the most to express their emotions. Figure 3 shows a
graphical representation of the number of responses
that we received. In this survey question, users were
allowed to select multiple answers, so that is why
the percentages do not add to one hundred.
Figure 3: H2: Facebook features usage frequency Vs
tendency to express emotions through them.
We investigated the effect of increase in the
number of likes received for one of the user’s posts
on the emotions of the social network users. As
shown in Figure 4, 81% of the sample showed that
the increase of the number of likes on their posts
affects their emotions positively. Only 19% reported
that the increase in the number of likes on their posts
does not affect their emotions.
Figure 4: H3: The effect of an increase in the number of
“likes” upon the emotions of social network user.
Facebook recognizes the relationships between
friends within the same social network. For example,
a friend can be a close friend, a family member or
general friend. We aimed at identifying the category
that has the most effect on the users of social
networks emotionally. Users could select more than
one answer for this question. Figure 5 shows that
majority of responses out of our sample tend to be
affected more by posts, comments, and likes from
close friends.
Figure 5: H4: How emotions are affected by different
types of social contacts.
The following figure shows how receiving a
Facebook gift from a friend within a social network
can affect the emotions of the user. 41% of the
sample expressed that their emotions will be affected
positively if they receive a Facebook gift. 59% of
the sample users showed that receiving a gift does
not affect their emotions. It also shows that 47% of
the sample’s emotions are affected positively if they
are invited to an event, such as birthdays or
weddings and 53% of the sample will not be affected
by such invitations. In the next section, we will
explain how we will use the study results to
automatically guess the social network users’
emotions.
Figure 6: H5 and H6: The impact of receiving a gift or
being invited to an event on emotions.
4 FUTURE WORK
We will explore the utilization of our study findings
to achieve more accurate emotion classification. We
will assign weights to status messages of the social
network users user based on the comments, degree
of connection between them and their friends and
the number of likes for each post. We will run set of
experiments with various weights and compare their
0
20
40
60
80
100
Status
upda…
Photos
Events
Private
Mess…
Notes
%
Feature
usage
Frequency
Tendency
toexpress
emotions
81%
19%
Yes
No
0
50
100
150
200
Responses
Posts,comments,
likesfromfamily
members
Posts,comments,
likesfromclose
friends
Posts,comments,
likesfrom
generalFacebook
friends
0
10
20
30
40
50
60
70
Gifts Events
%
Yes
No
PECCS2013-InternationalConferenceonPervasiveandEmbeddedComputingandCommunicationSystems
116

results to the real emotions of the users to see which
weight variation is closer to their real emotions.
Based on the results of these experiments we will
select our weighting criteria. These weighting
criteria will be used to enhance the accuracy of
eliciting the emotion of the social networks users.
5 CONCLUSIONS
In our study, we investigated the way Facebook
users utilize Facebook multimodal features, such as
comments, likes, and relationships between contacts
to express emotions. The results of our survey
indicate that not only do users express emotions on
Facebook, but their emotions are also affected by the
type of interaction happening. Social network users
tend to express their emotions through status updates
and comments more than other features. They are
affected by written exchange between users in the
form of status updates, comments, and also the likes
of their friends on their activities. The number of
likes to their posts tends to affect their emotions
positively. In specific, social network users are
affected more by emotions exhibited in their close
friends’ posts. These results match our hypotheses
H1 through H4. However, the results invalidated
hypotheses H5 and H6 as social networks users were
not impacted by the gifts that they receive from their
friends nor the events that they were invited to.
From these results, we have a better understanding
on how social network features and the information
they encompass can be used to automatically elicit
the emotions of their user.
REFERENCES
Conti, M., S. Das, C. Bisdikian, M. Kumar, L. Ni, A.
Passarella, G. Roussos, G. Troster, G. Tsudik, F.
Zambonelli (2012). Looking Ahead in Pervasive
Computing: Challenges and Opportunities in the Era
of Cyber-Physical Convergence, Pervasive and Mobile
Computing. 8 (1), 2-21.
Facebook (2012). Facebook statistics. Available:
https://www.facebook.com/press/info.php?statistics.
Last accessed 20th Jun 2012
Hancock, J. T., Gee, K., Ciaccio, K., & Lin, J. M.-hwah.
(2008). I’ m Sad You’reSad: Emotional Contagion in
CMC, 295-298.
Kramer, A. (2012) Ripples in the ocean: Emotional
contagion on Facebook. Society for Personality and
Social Psychology meeting. San Diego, CA.
Kramer, A. D. (2012). The Spread of Emotion via
Facebook. ACM. 767-70.
Kramer, A. (2010). An unobtrusive behavioral model of
gross national happiness. In Proc. CHI '10. ACM. 287-
290.
Walter, C., Saarlandes, U., Kipp, D. M. (2006). Artificial
Emotions.
Yassine, M., & Hajj, H. (2010). A Framework for
Emotion Mining from Text in Online Social
Networks. (2010). IEEE International Conference on
Data Mining Workshops, 1136-1142.
TowardsusingMultimodalFeaturesofSocialNetworksforImprovedContextualEmotionDetection
117