Sarcasm Detection Method to Improve Review Analysis
Shota Suzuki, Ryohei Orihara, Yuichi Sei, Yasuyuki Tahara and Akihiko Ohsuga
Graduate School of Information Systems, University of Electro-Communications, Tokyo, Japan
Keywords:
Natural Languages Processing, Opinion Mining, Sarcasm, Sentiment, Online Review.
Abstract:
Currently, classifying sarcastic sentences into positive and negative sentiments has been a difficult problem
and an important task. The sarcastic sentences could indicate negative meaning by using positive expressions,
or positive meaning by using negative expressions.Sarcasm is a special kind of sentiment that comprise of
words which mean the opposite of what you really want to say, especially in order to insult or wit someone, to
show irritation, or to be funny.Therefore, determining sarcasm is an important task in order to correctly classify
the sentence. In this paper, we propose an approach to detect sarcasm.First, we apply dependency parsing to
amazon review data. After that, we classify phrases in the sentence into the proposed phrase based on the
sequence of part-of-speech as proposed by Bharti et al. After being classified into either one of the phrase
types, it is determined whether each phrase is positive or negative.If the emotions of the situation phrases and
the sentiment phrases are different, the sentence is determined to be a “sarcasm”. Using the above method ,
the experimental result shows the effectiveness of our method as compared with the the existing research.
1 INTRODUCTION
With the spread of web services, everyone can post a
review of a product on the Web and it is readable from
anyone. The user review published on the web is a
valuable information resource to which another user
refers when she/he is making decision on a purchase.
However, selecting the useful information manually
to the user from the vast amount of review requires
much effort. Therefore, the study of current reputa-
tion analysis to automate the analysis of the text have
been conducted. In general, reputation analysis uses
polarity detection techniques that classify statements
into positive and negative ones. However, the current
polarity detection techniques only consider the emo-
tion of each word of the sentences. Thus it is diffi-
cult to correctly judge the polarity of expressions such
as sarcastic sentences that does not directly express
their intention. Sarcasm is a special type of sentiment
which plays a role as an interfering factor that can flip
the polarity of the given text. Given an example of
tweet: “Nothing I love more than a crowded library
with no seats #sarcasm”. Although this example uses
a word “love” to express the positive sentiment, the
tweet as a whole expresses negative sentiment toward
the library. Figure 1 shows a typical procedure of cur-
rent polarity detection techniques. In the left balloon
of Figure 1, although the text should be classified as
negative, it is mistakenly determined as positive by
With the spread of web services, everyone can post a
review of a product on the Web and it is readable from
anyone. The user review published on the web is a
valuable information resource to which another user
refers when she/he is making decision on a purchase.
However, selecting the useful information manually
to the user from the vast amount of review requires
much effort. Therefore, the study of current reputa-
tion analysis to automate the analysis of the text have
been conducted. In general, reputation analysis uses
arity detection techniques that classify statements
into positive and negative ones. However, the current
arity detection techniques only consider the emo-
tion of each word of the sentences. Thus it is diffi-
cult to correctly judge the polarity of expressions such
as sarcastic sentences that does not directly express
ƌĞǀŝĞǁƐĞƚ
ϭƌĞǀŝĞǁ
ŶĞƵƚƌĂů
ƉŽƐŝƚŝǀĞ
ŶĞŐĂƚŝǀĞ
㼟㼑㼚㼠㼕㼙㼑㼚㼠㻌㼟㼡㼙㼙㼍㼞㼕㼦㼍㼠㼕㼛㼚
tŽƌĚůĞǀĞůŽĨƚŚĞ
ƉŽůĂƌŝƚLJũƵĚŐĞŵĞŶƚĚŝĐƚŝŽŶĂƌLJ
ŽƵůĚŝŶĐůƵĚĞƐĂƌĐĂƐƚŝĐ
ƐƚĂƚĞŵĞŶƚƐƚŚĂƚƐŚŽƵůĚďĞ
ĐůĂƐƐŝĨŝĞĚĂƐŶĞŐĂƚŝǀĞ͘
Ğdž/ĂƉƉƌĞĐŝĂƚĞƚŚŝƐŐĂŵĞ
ďLJŵĂŬŝŶŐŵĞƐǁĞĂƌŶŽƚƚŽ
ďƵLJƚŚŝƐŐĂŵĞĂŐĂŝŶ͘
ŽƵůĚŝŶĐůƵĚĞƐĂƌĐĂƐƚŝĐ
ƐƚĂƚĞŵĞŶƚƐƚŚĂƚƐŚŽƵůĚďĞ
ĐůĂƐƐŝĨŝĞĚĂƐƉŽƐŝƚŝǀĞ͘
ĞdždŚŝƐŐĂŵĞŝƐƚŽŽŵƵĐŚ
ĂƚƚƌĂĐƚŝǀĞ͕ƐŽŝƚƚƌŽƵďůĞƐŵĞ͘
/ĂŵŐŽŝŶŐƚŽďĞĂũƵŶŬŝĞ͘
Figure 1: Typical procedure of current polarity detection.
Figure 1: Typical procedure of current polarity detection.
an existing polarity detector. Similarly, the detector
would determine as negative by mistake the text that
needs to be classified positively, as in the right bal-
loon of the Figure 1. As mentioned above, sarcastic
texts affect the classification accuracy of the polarity
detector.
In this paper, we propose a sarcasm-emotion de-
tection method for improvement of accuracy of emo-
tion determination intended for reviews. Our method
is based on Bharti’s method and determines whether
Suzuki S., Orihara R., Sei Y., Tahara Y. and Ohsuga A.
Sarcasm Detection Method to Improve Review Analysis.
DOI: 10.5220/0006192805190526
In Proceedings of the 9th International Conference on Agents and Artificial Intelligence (ICAART 2017), pages 519-526
ISBN: 978-989-758-220-2
Copyright
c
2017 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
519
a review in Japanese is a sarcasm. Since in Japanese
ambiguous expressions appear more frequently than
in English, we propose a method that can cope with
ambiguous expressions. Bharti et al. dealt with sar-
casm detection using Twitter. Bharti et al. proposed
phrases called situation phrases and sentiment phrases
to detect sarcasm. In this paper, situation phrases
and sentiment phrases are called proposed phrases
in total. Using the phrase above, Bharti et al. pro-
posed a PBLGA(a parsing-based lexical generation
algorithm) method for determining the sarcasm. For
more information about Bharti’s method refer to 2.1.
Bharti’s method first parses the tweets. Then it ana-
lyzes the clauses to detect proposed phrases in them.
If a phrase is detected as a situation phrase or a sen-
timent phrase, it phrases are stored the instance of
positive/negative sentiment/situation phrases. If pro-
posed phrases were determined a negative situation
phrase and a positive sentiment phrase, or a positive
situation phrase and a negative sentiment phrase, the
tweet is determined as sarcasm”. In Bharti’s method,
the instances of positive/negative sentiment/situation
phrases are stored in order to find sarcastic tweets
in a new corpus by means of simple string match
only. On the basis of the Bharti’s method, we pro-
pose a sarcasm-emotion detection method based on
polarities of words. We determine the emotion of
the proposed phrases based on the number of words
with the emotion included in the phrases just as the
Bharti’s method. From the results of counting, our
approach determines the emotions of the proposed
phrases. Further, if the numbers of P and N are the
same, we will determine the emotion of the proposed
phrases using the tf-idf scores. After judging the emo-
tion of the phrase, we determine the sarcastic state-
ment. Specifically, if the emotions of the proposed
phrase are different, we determine “sarcasm”. The
main difference between this research and Santoth et
al. research are shown in as follows.
We propose sarcasm-emotional detection method
based on emotion of phrase.
In order to improve the determinetion accuracy of
phrase emotion, feelings of phrases are selected
manually.
We conducted the evaluation experiment in order to
demonstrate the usefulness the proposed method. The
proposed method is implemented to analyze review
texts of computer games. When applied to 140 re-
views of a game “MSG EXTREME VS-FORCE”, the
proposed method could determine sarcastic sentences
with the precision of 0.79 and the recall of 0.56.
The structure of this paper is as follows. Section
2 surveys the existing research of sarcasm detection.
In Section 3, as an assumption of our research, we
discuss the concept of sarcastic sentence. Section 4
describes our proposed method. In Section 5, we eval-
uate how our proposed method can detect sarcasms in
comparison with the existing sarcasm detection meth-
ods. Section 6 discusses the results of the proposed
method and Section 7 concludes this paper.
2 RELATED WORK
In this section, we survey the existing research of sar-
casm detection.
2.1 Study in English Text in SNS Sites
Bharti et al. determined sarcasm in Twitter based
on two of their proposed method, namely, PBLGA
and IWS algorithm. In order to determine sarcasm in
Twitter data, PBLGA, or parsing based lexical gen-
eration algorithm, either one of the following combi-
nations must present in a tweet: (a) contradiction of
negative sentiment and positive situation (b) contra-
diction of positive sentiment and negative situation.
Bharti’s method first parses the tweets. Then it ana-
lyzes the clauses to detect proposed phrases in them.
If a phrase is detected as a situation phrase or a senti-
ment phrase, its sentiment score is determined based
on the number of positive and negative words in it. If
the sentiment score is positive, the proposed phrase
becomes positive, and if the sentiment score is nega-
tive, the proposed phrase becomes negative. If a tweet
contains a negative situation phrase and a positive
sentiment phrase, or a positive situation phrase and a
negative sentiment phrase, the tweet is determined as
sarcastic. In Bharti’s method, the instances of pos-
itive/negative sentiment/situation phrases are stored
in order to find sarcastic tweets in a new corpus by
means of simple string match only. IWS algorithm
is an algorithm to determine the sarcasm in view of
the interjection representation. If interjection appears
in the beginning of the tweet and intensifier appears
other than in the beginning, than a tweet has high
probability to classify as sarcastic.
As a preliminary of the experiment, dealing 50000
tweets with hash tags as training data in order to select
a phrase that becomes the emotion of the proposed
phrase. They constructed proposed phrases using the
training data. The test experiment was applying the
two algorithms perform two kinds experiment with-
out hash tag and hash tags into two types of data sets.
The result of applying the IWS approach with respect
to sarcasm tweets with the hash tag called #sarcasm
representing the sarcasm, obtained 96%of accuracy,
superiority of Bharti et al. Also, when they compared
ICAART 2017 - 9th International Conference on Agents and Artificial Intelligence
520
the two algorithms, IWS algorithm is a superiority
compared to PBLGA algorithm. Examples of each
emotion phrase data Bharti et al. have been used is as
shown in Table 1.
Table 1: Phrases used by Bharti et al.
positive sentiment phrase: Great, lucky, cute, good,
glad, delicious, best, Awesome, Perfect, joy, strong,
hilarious, better, goreous, honest, innocent, talented,
nice, happy, Pretty, proud, excellent
negative sentiment phrase: terrible, little, bit,
little, expensive, half, cold, food, tight jeans,
rude, wrong, dramatic, abusive, unhealthy, crap,
ugly, excuse, dirty, troubled, least, hard, bad,
bad, a few days
positive situation phrase: no regret, love,
love discovering, just love, effectively making,
absolutely Love, feel so loved, wanna be loved,
falling, honesty tell, will fly,
love seeing, winning, now enjoying managing
negative situation phrase: Clarifying, are pumped,
released, are arriving, babysitting, only run, kicking,
crashing, destroyed, attacking, criticizing, is lying,
confused, dividing, exhausted, keep arguing,
will be, gets stuck, is losing, biting, shouting
2.2 Sarcasm Extraction Method based
on Patterns of Evaluation
Expressions
Hiai et al.(Hiai et al.(2016)) propose a extraction
method of sarcastic sentences in prodect review. First,
they analyze sarcastic sentences in prodect reviews
and classify the sentences into 8 classes by focus-
ing on evaluation expressions. Next, they generate
classification rules for each class and use them to ex-
tract sarcastic sentences. Their method consists of
three stage; judgement processes based on reles for
8 classes, boosting rules and rejection rules. In the
experiment, they compare their method with a base-
line based on a simple rule. The experimental result
shows the effectiveness of their method. However,
they did not compare with other sarcasm determina-
tion method.
2.3 Automatic Detection of Sarcasm in
BBS Posts
Isono et al. (Isono et al.2013) propose two detec-
tion systems that determine sarcasm and slander in
posts on bullentim board system(BBS). They made a
corpus of sarcasm in BBS, and classified sarcasm in-
stances into eight classes: interrogative, guess, give-
up, unbalance, exaggeration, shock, metaphor, and
constract. For each sarcasm class, they constracted
syntactic patterns for detection of sarcasm that in-
clude sentence structures and polarity conditions of
the target sentence, the previous sentence and the next
sentence. Their first system detects sarcasm using a
database of the syntactic patterns. They made a cor-
pus of slander using Support Vector Machine(SVM),
where as features, they use frequencies of words in
the list, and positive expressions and negative expres-
sions in the target sentence, the previous sentence and
the next sentence. In the experiment, the proposed
systems can achieve superior F-measures compared
with baseline systems. But, the accuracy of the sys-
tem for determining the sarcasm was low.
3 TASK SETTINGS
In this section, we describe the definition and exam-
ples of “sarcastic statement”. Around the definition of
sarcasm, actively debate is carried out in psychology
and a wide range of fields. However, it cannot obtain
a clear answer what kind of linguistic phenomenon
sarcasm is (Utsumi reference).
In this study, we focus on sarcastic sentences that
cause errors in polarity determination. We define the
sarcastic statement as follows.
Hiai et al.(Hiai et al.(2016)) propose a extraction
method of sarcastic sentences in prodect review. First,
they analyze sarcastic sentences in prodect reviews
and classify the sentences into 8 classes by focus-
ing on evaluation expressions. Next, they generate
Definition of sarcasm
“Negative expression that convey a positive
meaning”or“Positive expression that convey a
negative meaning”
An example of sarcastic sentence corresponding
An example of sarcastic sentence corresponding
to “Negative expression that convey a positive mean-
ing” is as follows.
tract sarcastic sentences. Their method consists of
three stage; judgement processes based on reles for
8 classes, boosting rules and rejection rules. In the
experiment, they compare their method with a base-
line based on a simple rule. The experimental result
shows the effectiveness of their method. However,
they did not compare with other sarcasm determina-
Automatic Detection of Sarcasm in
Example 1
I bought a very tasty
cake on the way home
after I bought the game. I disposed
of the
game wrapped in wrapping paper of the cake!
Thank you!
Example 1 includes more positive words (with sin-
Example 1 includes more positive words (with sin-
gle underlines) than negative words (with double un-
derlines), while its overall meaning is negative. This
Example 1 tends to be classified as positive though
actually it expresses a negative feeling. Further, an
example of sarcastic sentence corresponding to “Pos-
itive expression that convey a negative meaning” is as
follows.
Sarcasm Detection Method to Improve Review Analysis
521
follows.
Example 2
This game is too much attractive
, so it
troubles me
. I am going to be a junkie.
Example 2 includes more negative words(with sin-
Example 2 includes more negative words(with sin-
gle underlines) than positive words(with double un-
derlines), while its overall sentiment is positive. This
Example 2 tends to be classified as negative though
actually it expresses a positive feeling. We identify
the sarcastic statement that intention and expression
are different. Based on the above definition, we deter-
mine the sarcasm in the next section.
4 PROPOSED METHOD
In this section, we describe in detail the proposed
method. On the basis of the Bharti’s method, we pro-
pose a sarcasm-emotion detection method based on
polarities on words. It describes the method determin-
ing the sarcasm or emotion in the rest of the section.
4.1 Flow of Sarcasm Extraction
Technique
We describe an overview of the proposed method.
Figure 2 shows an overview of the proposed method.
The following describes the detail of Figure 2.
As the first step, we carry out crawling from the
review to collect evaluation data. As the second
step, we parse the review and break down the whole
sentence into phrases. As the third step, we judge
whether the proposed phrase in the parsed ones are
included. In this study, we propose situation phrase
and sentiment phrase as our proposed phrase. In ad-
dition, we define the proposed phrase as follows.
dition, we define the proposed phrase as follows.
Definition of proposed phases
We call sentiment phrases and situation phrases
as proposed phrases in total.
For the parsed ones corresponding to the proposed
For the parsed ones corresponding to the proposed
phrase, we manually determine the emotion (positive
or negative) in the parsed ones. When judging emo-
tion of parsed ones, we create the dictionary manu-
ally. As a final step, we judge the sarcasm-emotion
detection by using the proposed phrases. By count-
ing the parsed ones to the positive or negative, we
determine the feeling of the proposed phrases. We
detect whether the review is sarcastic by using the
feeling of the proposed phrases. Sarcasm-emotion de-
tection is described in detail in 4.3. We applied the
proposed method to the review of the online review
site. In the rest, we describe in detail with respect to
the proposed method. Preprocessing before sarcasm-
emotion detection is described in 4.2. Section 4.3 de-
scribes sarcasm-emotion detection method to detect
sarcasm.
ϳ
ƌĞǀŝĞǁ
ĚĂƚĂƐĞƚ
ĐƌĂǁůŝŶŐ
ĚĞƉĞŶĚĞŶĐLJƉĂƌƐŝŶŐ
ƉŚƌĂƐĞ
ƉŚƌĂƐĞ
ƉŚƌĂƐĞ
ƉŚƌĂƐĞ
ƐŝƚƵĂƚŝŽŶƉŚƌĂƐĞ ƐĞŶƚŝŵĞŶƚƉŚƌĂƐĞ
ũƵĚŐĞǁŚĞƚŚĞƌ
ƉƌŽƉŽƐĞĚƉŚƌĂƐĞ
ƉŽƐŝƚŝǀĞŽƌŶĞŐĂƚŝǀĞ
ƐĂƌĐĂƐŵͲĞŵŽƚŝŽŶĚĞƚĞĐƚŝŽŶ
Figure 2: The proposed method.
4.2 Phrase Analysis
Here, we describe preprocessing for sarcasm-
emotional detection. Specifically, it is judged and
counted whether emotions are included in the parsed
ones. After crawling the review, we apply the depen-
dency parsing for a review. The dependency pars-
ing is a technique to analyze the syntactic relation-
ship, such as relationships between words to isolate
the whole sentence to morpheme. In this research, we
used Cabocha. CaboCha is a Japanese dependency
analyzer based on support vector machines. Figure 3
shows an example which carry out the syntax analysis
using Cabocha.
The following sentence is the input of the dependency
parsing shown in Figure 3. “This game is very inter-
esting. I threw it from the top floor of the momentum
left over by Abenoharukasu. Thanks to alchemist who
has drilling this kind of terrible game!!!. Thank you
Bannam. From a result of the dependency parsing,
we made a review the set of the parsed ones. Next,
we disucuss the dependency relation. In Figure 3,
1
corresponds to “threw”, and
2
corresponds to “from
ICAART 2017 - 9th International Conference on Agents and Artificial Intelligence
522
ϰ
䛸䛶䜒
㠃ⓑ䛔䛷䛩
ໃ䛔
వ䛳䛶
䛒䜉䛾䝝䝹䜹䝇䛾
᭱ୖ㝵䛛䜙䐠
ᢞ䛢ᤞ䛶䜎䛧䛯䐟
䛣䜣䛺
䜽䝋䝀䞊䜢
䝞䞁䝘䝮䛥䜣䛒䜚䛜䛸䛖͊͊
㘐ᡂ䛧䛶䛟䜜䛯
㘐㔠⾡䛯䛱䛻
ឤㅰ͊͊͊
Figure 3: Dependency parsing.
the top floor”. Considering the analysis result of Fig-
ure 3, it turns out that
2
qualifies
1
.
Using the syntax analysis, we determine whether
the part of speech of the parsed ones can be included
as the part of speech of the proposed phrases. As a
proposed phrases, we propose a situation phrase and
a sentiment phrase as the type of phrase. Situation
phrase means to the “action” in the sentence, and sen-
timent phrase means to the “emotion” in the sentence.
Proposed phrases is represented by a combination of
parts of speech. Table 2 shows a part of speech corre-
sponding to the proposed phrases.
Table 2: Correspondence table of the phrases.
We treat a parsed ones as a situation phrase or sen-
timent phrase according to which combination of the
parts of speech given in Table 2 matches the target
phrase. Table 3 shows the result of sorting those ap-
plicable to proposed phrase from the parsed ones in
Figure 3. It should be noted that in this paper is the
selection of the phrase to depend on parsing the re-
sults of Cabocha. Also, if it contains proper nouns in
the phrase, the phrase is treated as noise representa-
tion.
Finally, we determine the phrase that can be deter-
mined include feelings to phrases correspond to the
proposed phrase (including positive or negative). In
this study, we determined whether positive or neg-
Table 3: Results of phrase classification of the example.
sentiment phrase: interesting,
the top floor, told me to drilling,
alchemy, thanks
situation phrase: left over, threw,
terrible
ative manually. Table 4 shows the result of emo-
tional sorting those applicable to proposed phrases
from the phrase in Figure 3. Incidentally, we deal with
Table 4: Emotional phrase correspondence table.
Positive sentiment phrase : thanks, interesting
Negative sentiment phrase : None
Positive situation phrase : None
Negative situation phrase : threw
phrases that include proper names as noise representa-
tion. From the above procedure, we calculate whether
contains a number of emotions subject to review. By
using a number of emotions subject to review, we de-
termine the feelings of phrase. In the next section,
we describe the method of calculating the phrase of
emotion and the sarcasm-emotion detection.
4.3 Sarcasm-Emotion Detection
Here, we describe a method sarcasm-emotion detec-
tion by using the emotion of the proposed phrases.
In the previous section, we examined whether or not
there is a phrase that can be judged as positive or
negative for the phrase that corresponds to the pro-
posed phrases. Here, we describe about the judges
approach the feelings of the “proposed phrase”. After
determiningthe feeling of proposed phrases, we judge
sarcasm-emotion detection method as a final evalua-
tion.
4.3.1 Deciding Sentiment of a Proposed Phrase
As a prestage of sarcasm-emotion detection, we eval-
uated the feelings of the phrase. In order to assess
the feeling of the “proposed phrases”, we define the
following formula.
PR =
PWP
TWP
NR =
NWP
TWP
SentimentScore = PR NR
For the proposed phrases, PR is positive ratio, NR is
negative ratio, TWP indicates the number of all of the
Sarcasm Detection Method to Improve Review Analysis
523
proposed phrase with respect to each reviews. In ad-
dition, PWP and NWP are the numbers of positive
and negativeproposed phrases respectively. Using the
above equations, we describe technique for determin-
ing the emotion of the phrases below. Procedure for
determining the emotion of the proposed phrases is as
shown in Figure 4. Let us explain Figure 4.
^ĞŶƚŝŵĞŶƚ^ĐŽƌĞŽĨĂƉƌŽƉŽƐĞĚƉŚƌĂƐĞ Ͳǫ
dŚĞĞŵŽƚŝŽŶŽĨƚŚĞƉŚƌĂƐĞ
/ƐĚĞƚĞƌŵŝŶĞĚďLJƵƐŝŶŐƚŚĞƚĨͲŝĚĨ;ϮͿ
/Ĩ^ĞŶƚŝŵĞŶƚ^ĐŽƌĞфϬїEĞŐĂƚŝǀĞ
/Ĩ^ĞŶƚŝŵĞŶƚ^ĐŽƌĞхϬїWŽƐŝƚŝǀĞ;ϭͿ
zĞƐ
^ĂƌĐĂƐŵͲĞŵŽƚŝŽŶĂůĚĞĐŝƐŝŽŶ
ĞĐŝĚŝŶŐƐĞŶƚŝŵĞŶƚŽĨĂƉƌŽƉŽƐĂůƉŚƌĂƐĞ
Figure 4: Emotion detection of proposed phrases.
Figure 4: Emotion detection of proposed phrases.
If the SentimentScore of the proposed phrases is
unequal to 0, as in (1) of the Figure 4, then the emo-
tion of the phrase is determined based on the Senti-
mentScore, namely, positive when SentimentScore >
0 and negativeotherwise. If the SentimentScore of the
proposed phrases is equal to 0, as in (2) of the Figure
4, then the emotion of the phrase is determined us-
ing tf-idf. The tf-idf is a method of calculating the
weight of a particular word in the document that by
combining tf method and idf method. This technique
has been utilized in the field of information retrieval
and text mining and machine learning. It is shown
below with respect to the meaning of tf method and
idf method. tf (term frequency) represents term fre-
quency, it represents the number of times a particu-
lar word is found at the text in. idf (inverse docu-
ment frequency) represents the natural logarithm of
the inverse of the number of documents that contain
the words in the training data. A high frequency in
the text it is seen that as much the word is important.
Calculation formula for weighting is as formula.
w
t,d
= t f
t,d
log
N
d f
t
Here, N is the data for the entire review data. Further,
t f
t,d
in the number oft appears in a document d, d f
t
is
the number of documents in which the word t appears
in the N sentences. By using the above determination
process, it was evaluated in the emotions of the sit-
uation phrase and sentiment phrase. Taking Table 4
as an example, the emotion of the sentiment phrase
is determined to be positive, emotion of the situation
phrase is determined to be negative. After determin-
ing the feelings of the proposed phrase, we will make
a final sarcasm-emotion detection.
4.3.2 Deciding Sarcasm-Emotion Detection of
Reviews
Next, we determine the sarcasm or emotion by using
the emotion of the proposed phrase. Figure 5 shows
sarcasm-emotion detection method.
Figure 5: Sarcasm-emotion detection.
Based on the definition, it determines that the sar-
castic statement in this study if the emotions of the
proposed phrases are different. Taking Table 4 as an
example, the emotion of sentiment phrase is judged
to be a positive, the emotion of the situation phrase is
determineded to be a negative, therefore this review is
determined to be the sarcastic statement. Also, if both
of the emotion of the situation phrase and sentiment
phrase are the same, it is determined that the target
sentence is non-sarcastic statement, the same emotion
of the entire sentence. By using the sarcasm-emotion
detection method described above, we determine the
sarcasm.
5 SARCASM JUDGEMENT
EVALUATION
In this section, we confirm the accuracy of the pro-
posed method. At that time, we compared to the ex-
isting sarcasm detection method.
5.1 Experimental Set-up
We conduct an experiment using reviews of a product
that are “flaming”, that is, including many negative
reviews. We evaluate as a target star5 reviews of Mo-
bile Suit Gundam EXTREME VS-FORCE of amazon
products. The subject of the review has been “flam-
ing” for game content is unpopular. Because there
are many negative expressions in spite of high scores,
we treate as experimental data in this experiment. We
applied the proposed method to 140 reviews of star5
reviews. Before applying the approach, we labeled
“sarcastic statement or non-sarcastic statement” and
“positive or negative or neither”. Determination of
ICAART 2017 - 9th International Conference on Agents and Artificial Intelligence
524
sarcasm is based on the definition given in section
3. Also, if you cannot determine whether positive or
negative in the non-sarcastic statement, we treate as
“neither”. Examples of non-sarcasm statement nei-
ther” are included those such as the following.
ther” are included those such as the following.
Example3
I want to say about this game only the following
words: May the Force be with you.
As mentioned above, we dealt things not estab-
lished as sentences as “neither”. Table 5 and Table 6
show the result of labeling.
Table 5: Labeling result 1.
sarcastic statement 89
non-sarcastic statement 51
Table 6: Labeling result 2.
positive 33
negative 11
neither 7
Table 5 is a result of the labeling process to ei-
ther the entire review “sarcastic statement” or “non-
sarcastic statement”. Table 6 is a result of the labeling
process in three of emotion is non-sarcastic state-
ment”. Result of labeling, 89 reviews from 140 re-
views is determined to be sarcastic statement. Fur-
ther, we can assume that “flaming reviews” are more
likely to contain sarcastic in the star5 reviews. Using
the above results, we perform each experiment.
5.2 Evaluation Method
Here, we describe the experimental results of the pro-
posed method. To compare the results with exist-
ing research, three parameters are considered, namely,
precision, recall and f-score. We describe below for
each evaluation method. Table 7 shows evaluation
method in this research.
Table 7: Evaluation method.
fact
sarcasm non-sarcasm
prediction sarcasm T
p
F
p
@non-sarcasm F
N
T
N
We write the following precision , recall , F-score by
using the evaluation method in Table 7.
Precision: Among the predicted data, actually shows
the proportion of what is positive. Formula becomes
(1)
Precision =
T
p
T
p
+ F
p
(1)
Recall: From what is positive, indicating the percent-
age is expected to be positive. Formula becomes (2)
Recall =
T
p
T
p
+ F
N
(2)
F-score: Show the harmonic mean of precision and
recall.Formula becomes(3)
F score =
2· Precision· Recall
Precision+ Recall
(3)
Using the above evaluation method, we evaluate the
accuracy of the proposed method.
5.3 Evaluation Results
We compare with the existing sarcastic statement de-
tection method and the proposed method in this data
set. We apply to this data set the sarcasm detection
method of Isono et al. (2013) and Hiai et al. (2016).
Table 8 shows the existing sarcastic statement detec-
tion method and the proposed method comparison re-
sult.
The result of applying the proposed method to the
data set, proposed method achieved 0.79, 0.56, and
0.63 precision, recall and f-score respectivelyin Ama-
zon dataset. As the result of applying the existing sar-
casm determination method in this data set, it found
that the proposed method outperformed the existing
method in terms of the accuracy to determine sarcas-
tic statements. The reason of this result would be be-
cause the expressions specific to games did not match
the syntactic patterns of the existing research. There-
fore we make clear that the advantage of our proposed
method is due to no use of syntactic patterns. From
the above, determining the sarcastic statement by us-
ing the our proposed method is useful method in the
online revies site.
Table 8: Experiment result 1.
precision recall F score
proposed method 0.79 0.56 0.63
Hiai et al. 0.60 0.03 0.06
Isono et al. 0.63 0.07 0.13
6 DISCUSSIONS
From the experimental result, it shows the results
of high accuracy of proposed method for sarcasm
Sarcasm Detection Method to Improve Review Analysis
525
of our definition. However, they also show the re-
sults in which our method sometimes determine Sen-
timentScores that the reviews do not actually mean
if the numbers of positive and negative phrases are
much different. It will be discussed because our pro-
posed method depends on the numbers of the pro-
posed phrases. Therefore, we consider that it can not
determine the sentences as “sarcasm” such as follow-
ing.
ing.
Example 4
It is like a dream that I can use such a cool minia-
ture aircraft so early by purchasing such a terri-
ble game!
A result of the proposed method apply to the ex-
ample 4, the proposed phrase determined “positive is”
by containing many positive phrase. Thus our method
determined that this review praised the product posi-
tively. This result shows that there are sarcastic sen-
tences that cannot be correctly classified by counting
the phrases in the reviews. In order to judge this re-
view as sarcastic, we will need to consider combina-
tions of phrases or interjections.
7 CONCLUSION
In this paper, we proposed a sarcastic statement deter-
mination for the classification accuracy improvement
of emotional judgment in the online review sites. The
result of the comparison of the existing research, we
could confirm the usefulness of the proposed method.
However, there are sarcastic reviews our method can-
not judge as sarcastic may be because our method de-
tect sarcasm on the basis of the numbers of positive
and negativephrases. In order to determine the review
such as described above, we will need to improve our
method. We are going to improve our method to in-
crease the accuracy of sarcasm and emotion detection
by incorporating interjection words and considering
combinations of phrases.
ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant
Numbers 26330081, 26870201, 16K12411. In per-
forming this study, we would like to thank everyone
that has helped us.
REFERENCES
Santoth Kumar Bharti , Korra Sathya Babu , Sanjay Kumar
Jena. Parsing-based Sarcasm Sentiment Recognition
in Twitter Data, The 2015 IEEE/ACM International
Conference on Advances in Social Networks Analysis
and Mining
Ashwin Rajadesingan , Reza Zafarani , Huan Liu. Sarcasm
Detection on Twitter: A Behavioral Modeling Ap-
proach, Proceedingsof the Eighth ACM International
Conference on Web Search and Data Mining, 2015
Dmitry Davidov , Ari Rappoport . Semi-supervised recog-
nition of sarcastic sentences in Twitter and Amazon,
Proceedings of the Fourteenth Conference on Compu-
tational Natural Language Learning, 2010
F. Barbieri, H. Saggion, and F. Ronzano . Modeling sarcasm
in twitter , a novel approach, in Association for Com-
putational Linguistics, 2014
Fumiya Isono , Suguru Matsuyoshi , Fumiyo Fukumoto
. Automatic Detection of Sarcasm in BBS Posts
Based on Sarcasm Classification, Information Pro-
cessing Society of Japan SIG Technical Report, 2013
in Japanese
Satoshi Hiai , Kazutaka Shimada. Analysis and Detection
of Sarcastic Sentences Focused on Evaluative Expres-
sions, Information Processing Society of Japan SIG
Technical Report, 2016 in Japanese
Akira Utsumi , A Cognitive and Computational Model of
Irony Interpretation, Transactions of Information Pro-
cessing Society of Japan, Vol41 , No9, pp.2498-2509,
2000 in Japanese.
CaboCha , http://chasen.org/˜taku/software/cabocha/
SentimentAnalyZer, http://www.ohsuga.is.uec.ac.jp/
sentiment
ICAART 2017 - 9th International Conference on Agents and Artificial Intelligence
526