Identifying Tweets that Contain a ’Heartwarming Story’
Manabu Okumura
1
, Yohei Yamaguchi
1
, Masatomo Suzuki
2
and Hiroko Otsuka
2
1
Tokyo Institute of Technology, 4259 Nagatsuta, Midori, Yokohama 226-8503, Japan
2
Future University Hakodate, 116-2 Kamedanakano-cho, Hakodate-shi, Hokkaido 041-8655, Japan
Keywords:
Social Media, Twitter, Emotion Analysis, Readers’ Emotion.
Abstract:
We present a rather new task of detecting and collecting tweets that contain heartwarming stories from a huge
amount of tweets on Twitter in this paper. We also present a method for identifying heartwarming tweets. Our
prediction method is based on a supervised learning algorithm in SVM along with features from the tweets.
We found by comparing the feature sets that adding sentiment features mostly improves the performance.
However, simply adding the features for detecting a story in a tweet (past tense and tweet length) cannot
contribute to improving the performance, while adding all the features to the baseline feature set mostly yields
the best performance from among the feature sets.
1 INTRODUCTION
Since users of social media such as blogs and Twit-
ter have been increasing with the development of the
Internet, we now have a huge amount and variety of
information on the Web. It is now very common to
effectively use such a vast amount of information dis-
seminated from many people worldwide to find out
their opinions and feelings. Therefore, the research
topics, sentiment analysis, and opinion mining, have
recently received more attention(Pang and Lee, 2008;
Liu, 2012).
Although a lot of work have recently been con-
ducted on sentiment analysis and opinion mining,
most of the work specifically targeted the writer’s
opinions and feelings. However, only a given few
came from the reader’s perspective. We believe sen-
timent analysis and opinion mining from the reader’s
perspective is also useful in many applications. When
we can devise a method for identifying what docu-
ments make readers happy, we will then be able to
more effectively mine a collection of ’heartwarming
stories. When we can identify what sentences or ex-
pressions make readers feel unpleasant, we will be
able to develop a system of supporting writing that
prevents writers from using rude expressions.
We were able to find the following few work
on sentiment analysis from the reader’s perspective.
Yang et al.(Yang et al., 2009) tried to construct
not only a writer-emotion corpus but also a reader-
emotion corpus. Furthermore, they tried to statisti-
cally analyze the corpus. Lin et al.(Lin et al., 2008)
built a reader-emotion classifier that classifies a doc-
ument into one of eight reader emotion classes. They
used Support Vector Machines (SVM) and the fol-
lowing types of features: character bigrams, words,
affix similarities, and emotional words. Hasegawa et
al.(Hasegawa et al., 2013) tried to predict the emotion
of the addressee and to generate a response that elicits
a specific emotion in the addressee’s mind. They tar-
geted Japanese Twitter posts as a source of dialogue
data and automatically built training data for learning
the predictors and generators.
Offensive content detection is not a new field,
but there are only a small number of existing work.
Razavi et al.(Razavi et al., 2010) treated the offen-
sive language detection problem as a machine learn-
ing task and adopted three-level classifications with
bag-of-words features that are based on an abusive ex-
pression dictionary.
In this paper, we first present a rather new task of
detecting and collecting documents that contain heart-
warming stories from a huge variety of contents on the
Web. More specifically, since we target tweets on the
social networking site Twitter(https://twitter.com/),
our task is to detect tweets that contain heartwarm-
ing stories from the huge amount of tweets posted
on Twitter. We then present a method for identifying
heartwarming tweets. We believe the task for detect-
ing documents (tweets) containing heartwarming sto-
ries is new in that they should not only make readers
happy (have positive sentiment from the reader’s per-
spective) but contain the writer’s actual experiences.
323
Okumura M., Yamaguchi Y., Suzuki M. and Otsuka H..
Identifying Tweets that Contain a ’Heartwarming Story’.
DOI: 10.5220/0005129503230326
In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval (KDIR-2014), pages 323-326
ISBN: 978-989-758-048-2
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
2 TWEETS THAT CONTAIN
HEARTWARMING STORIES
There have been many sites on the Web that are dedi-
cated to heartwarming stories
1
. Those stories on these
sites tend to touch the readers’ hearts. The visitors of
the sites will find a lot of joy in these stories. Heart-
warming stories are said to be the stories that will
make readers laugh, cry, and smile. However, the sto-
ries on these sites are manually collected.
Our task is to detect tweets that contain such heart-
warming stories from a huge amount of tweets on
Twitter. The following is a sample tweet that is con-
sidered heartwarming
2
:
6B.M®o/ÆùL
*(S3+8ÎVS.J*
#&I#C>.K+#(
7
(I was really tired. I stopped at the conve-
nience store because I could not make a boxed
lunch today. I was woken up by the staffs ca-
sual words of, ’Have a nice day.’)
6ÏÚ(@#&¯>#).
M'&SGS×#&I>
$& #VSª7
(I thought I missed the last train when I fell
down just in front of the closing door but the
st aff kindly opened the door again. He was
kind.)
3 COLLECTING
HEARTWARMING TWEETS
We tried to construct a corpus of heartwarming
tweets. Since it is difficult to collect heartwarming
tweets from the vast number of topics on Twitter,
we decided to construct a corpus of heartwarming
tweets that contain episodes concerning railway and
convenience-store staffs. We chose these two groups
of individuals because we often come in daily con-
tact with them as customers. We selected two railway
companies and a convenience store chain in Japan.
We first retrieved a collection of tweets from Twit-
ter that contained the following query: Company
name + Keywords indicating company staffs
3
. We
then tried to automatically remove the following types
of tweets:
1
For example, http://www.heartwarmingstories.net/.
2
We manually rephrased original tweets more formally.
3
For example, 6;7 (conductor), 6V7 (station
staff), 6ÎV7(store staff), and so on.
tweets created by bots,
retweets, and
tweets considered news articles.
Next, we manually judged whether each tweet
contained a heartwarming story by actually reading it.
The statistics of the corpus are shown in Table 1. As
seen in this table, the ratio of heartwarming tweets is
quite small, and therefore, it is challenging to identify
them from a complete collection of tweets.
4 DETECTING HEARTWARMING
TWEETS
Our method for detecting heartwarming tweets from a
collection of tweets is based on a classification model
trained by using SVM. We used SVM because the
learning algorithms have been successfully used in
text classification in the past.
The simplest way to construct a model for detect-
ing heartwarming tweets using supervised learning al-
gorithms is to use a ’bag of words’ (BoW) whose
parts-of-speech are a verb, a noun, and a suffix in the
tweet as the features. We used suffixes because ben-
eficiary expressions such as 6&LK7 and 6&
BI7 are suffixes.
However, we took into account the following ideas
and devised better features to improve the predic-
tion accuracy, because heartwarming tweets should
not only contain positive sentiments from the reader’s
perspective but also contain the actual experiences of
the writer:
1. When writers describe their positive feelings in a
tweet, readers might empathize and thus become
happy. Therefore, we used the lexica of polarity-
bearing words and calculate the number of posi-
tive and negative words as features.
We used two sentiment lexica as our lexical re-
source for the polarity-bearing words. The first
is available on the Web
4
. The lexicon was con-
structed using the method developed by Takamura
et al.(Takamura et al., 2005). The second was con-
structed using the method developed by Suzuki et
al.(Suzuki et al., 2007).
2. Since heartwarming tweets should also contain
actual experiences, clues for whether a tweet con-
tains a story should be incorporated. Therefore,
we used the length of the tweet and whether or
not the tweet contained verbs in the past tense as
features
5
.
4
http://www.lr.pi.titech.ac.jp/takamura/pndic en.html
5
Tweets describing a story might be long.
KDIR2014-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval
324
Table 1: Statistics of heartwarming tweets corpus.
Company Total No. Heartwarming Non-heartwarming
of tweets tweets tweets
Family-Mart 691 71 620
JR 657 48 609
Tokyo Metro 664 74 590
3. As noted in Table 1, our data set is severely im-
balanced and the heartwarming tweets class has
fewer examples than the other. Learning algo-
rithms that do not take into account the class im-
balance tend to be overwhelmed by the majority
class and ignore the minority one, and the over-
all accuracy of conventional learning algorithms
will thus significantly degrade, since their classi-
fiers are greatly biased towards the majority class.
Therefore, under-sampling (Zhang et al., 2010) is
used and the training is performed using an equal
number of positive and negative examples.
The sampling came from a class of methods that
alters the size of the training sets. Under-sampling
is an imbalanced data learning method that uses
only a subset of the majority class examples.
Under-sampling changes the training sets by ran-
domly sampling the examples from the major-
ity class training set and making it smaller. The
level of imbalance is reduced, with the hope that a
more balanced training set can provide better re-
sults. Among the various class-imbalance learn-
ing methods, under-sampling has been commonly
used. The under-sampling method we adopted
clusters the examples in the majority class first
and then randomly selects examples from these
clusters.
5 EXPERIMENTS
In this section, we report on the experimental results
using our prediction method described in the previous
section on our data collection described in Section 3.
We used the standard precision and recall for the
positive class of heartwarming tweets, and the F1
measure, which is the harmonic mean between the
precision and recall, to evaluate the prediction. We
used the libsvm implementation (Fan et al., 2005)
for SVM with the linear kernel in the 10-fold cross-
validation.
We compared the following different feature sets
for the prediction:
1. BoW whose parts-of-speech are a verb, a noun,
and a suffix in the tweet (baseline feature set),
Table 2: Prediction performance.
Family-Mart:
Feature Precision Recall F1-measure
Baseline 0.534 0.775 0.632
+ polarity 0.518 0.817 0.634
+ tense 0.529 0.775 0.629
+ length 0.417 0.845 0.558
All 0.378 0.789 0.511
JR:
Feature Precision Recall F1-measure
Baseline 0.183 0.625 0.283
+ polarity 0.208 0.729 0.324
+ tense 0.172 0.729 0.278
+ length 0.163 0.792 0.270
All 0.284 0.646 0.395
Tokyo Metro:
Feature Precision Recall F1-measure
Baseline 0.384 0.757 0.510
+ polarity 0.341 0.757 0.471
+ tense 0.364 0.797 0.500
+ length 0.290 0.784 0.423
All 0.425 0.838 0.564
2. BoW + polarity information (Please see 1. in Sec-
tion 4),
3. BoW + past tense (Please see 2. in Section 4),
4. BoW + tweet length (Please see 2. in Section 4),
5. BoW + polarity + past tense + tweet length + other
features (all features).
’Other features’ in the above feature sets indicate
whether interjections or quotation marks are in the
tweet. The experiments were performed using under-
sampling for all the feature sets.
In Table 2, we show the prediction results for the
three companies mentioned in Section 3 using our
method. The best performance is shown in boldface
for each company.
We found by comparing the feature sets that
adding sentiment features mostly improves the per-
formance. However, simply adding the features for
detecting a story in a tweet (past tense and tweet
length) cannot contribute to improving the perfor-
mance, while adding all the features to the baseline
feature set mostly yields the best performance from
among the feature sets.
IdentifyingTweetsthatContaina'HeartwarmingStory'
325
6 CONCLUSIONS
In this paper, we first presented a rather new task
of detecting and collecting tweets that contain heart-
warming stories from a huge amount of tweets on
Twitter. We then presented a method for identify-
ing heartwarming tweets. Our prediction method is
based on a supervised learning algorithm in SVM
along with the features from the tweets.
We need to improve the prediction performance
by devising more intelligent features for future work.
Furthermore, we also need to construct larger corpus
of heartwarming tweets for the experiments. Trying
the task in other languages such as English is also our
future work.
ACKNOWLEDGEMENTS
This work is conducted under a joint project with the
Survey Research Center Co. Ltd. and The Insti-
tute of Behavioral Sciences in Japan. We would like
to thank the following co-workers for their valuable
comments and suggestions: Hiroshi Tomatsuri, Kiy-
ohisa Takano, Hiroharu Ochihara, Yuichi Mohri, and
Hiromi Ito.
REFERENCES
Fan, R., Chen, P., and Lin, C. (2005). Working set selection
using second order information for training support
vector machines. The Journal of Machine Learning
Research.
Hasegawa, T., Kaji, N., and Yoshinaga, N. (2013). Pre-
dicting and eliciting addressee’s emotion in online di-
alogue. In Proceedings of the 51st Annual Meet-
ing of the Association for Computational Linguistics
(ACL2013), pages 964–972.
Lin, K. H.-Y., Yang, C., and Chen, H.-H. (2008). Emotion
classification of online news articles from the reader’s
perspective. In Proc. of the 2008 IEEE/WIC/ACM In-
ternational Conference on Web Intelligence (WI’08).
Liu, B. (2012). Sentiment Analysis and Opinion Mining.
Morgan & Claypool Publishers.
Pang, B. and Lee, L. (2008). Opinion mining and sentiment
analysis. Foundations and Trends in Information Re-
trieval, 2(1-2):1–135.
Razavi, A. H., Inkpen, D., Uritsky, S., and Matwin, S.
(2010). Offensive language detection using multi-
level classification. In Proceedings of the 23rd Cana-
dian conference on Advances in Artificial Intelligence
(AI’10), pages 16–27.
Suzuki, Y., Takamura, H., and Okumura, M. (2007). Semi-
supervised learning to classify evaluative expressions
from labeled and unlabeled texts. IEICE Transactions
on Information and Systems, pages 1516–1522.
Takamura, H., Inui, T., and Okumura, M. (2005). Extract-
ing semantic orientations of words using spin model.
In Proceedings of the 43rd Annual Meeting of the As-
sociation for Computational Linguistics (ACL2005),
pages 133–140.
Yang, C., Lin, K. H.-Y., and Chen, H.-H. (2009). Writer
meets reader: Emotion analysis of social media from
both the writer’s and reader’s perspectives. In Proc.
of the 2009 IEEE/WIC/ACM International Conference
on Web Intelligence (WI’09), pages 287–290.
Zhang, Y., Zhang, L., and Wang, Y. (2010). Cluster-based
majority under-sampling approaches for class imbal-
ance learning. In Proceedings of the 2nd IEEE In-
ternational Conference on Information and Financial
Engineering (ICIFE), pages 400–404.
KDIR2014-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval
326