Identifying Tweets that Contain a ’Heartwarming Story’

Manabu Okumura

, Yohei Yamaguchi

, Masatomo Suzuki

and Hiroko Otsuka

Tokyo Institute of Technology, 4259 Nagatsuta, Midori, Yokohama 226-8503, Japan

Future University Hakodate, 116-2 Kamedanakano-cho, Hakodate-shi, Hokkaido 041-8655, Japan

Keywords:

Social Media, Twitter, Emotion Analysis, Readers’ Emotion.

Abstract:

We present a rather new task of detecting and collecting tweets that contain heartwarming stories from a huge

amount of tweets on Twitter in this paper. We also present a method for identifying heartwarming tweets. Our

prediction method is based on a supervised learning algorithm in SVM along with features from the tweets.

We found by comparing the feature sets that adding sentiment features mostly improves the performance.

However, simply adding the features for detecting a story in a tweet (past tense and tweet length) cannot

contribute to improving the performance, while adding all the features to the baseline feature set mostly yields

the best performance from among the feature sets.

1 INTRODUCTION

Since users of social media such as blogs and Twit-

ter have been increasing with the development of the

Internet, we now have a huge amount and variety of

information on the Web. It is now very common to

effectively use such a vast amount of information dis-

seminated from many people worldwide to ﬁnd out

their opinions and feelings. Therefore, the research

topics, sentiment analysis, and opinion mining, have

recently received more attention(Pang and Lee, 2008;

Liu, 2012).

Although a lot of work have recently been con-

ducted on sentiment analysis and opinion mining,

most of the work speciﬁcally targeted the writer’s

opinions and feelings. However, only a given few

came from the reader’s perspective. We believe sen-

timent analysis and opinion mining from the reader’s

perspective is also useful in many applications. When

we can devise a method for identifying what docu-

ments make readers happy, we will then be able to

more effectively mine a collection of ’heartwarming

stories.’ When we can identify what sentences or ex-

pressions make readers feel unpleasant, we will be

able to develop a system of supporting writing that

prevents writers from using rude expressions.

We were able to ﬁnd the following few work

on sentiment analysis from the reader’s perspective.

Yang et al.(Yang et al., 2009) tried to construct

not only a writer-emotion corpus but also a reader-

emotion corpus. Furthermore, they tried to statisti-

cally analyze the corpus. Lin et al.(Lin et al., 2008)

built a reader-emotion classiﬁer that classiﬁes a doc-

ument into one of eight reader emotion classes. They

used Support Vector Machines (SVM) and the fol-

lowing types of features: character bigrams, words,

afﬁx similarities, and emotional words. Hasegawa et

al.(Hasegawa et al., 2013) tried to predict the emotion

of the addressee and to generate a response that elicits

a speciﬁc emotion in the addressee’s mind. They tar-

geted Japanese Twitter posts as a source of dialogue

data and automatically built training data for learning

the predictors and generators.

Offensive content detection is not a new ﬁeld,

but there are only a small number of existing work.

Razavi et al.(Razavi et al., 2010) treated the offen-

sive language detection problem as a machine learn-

ing task and adopted three-level classiﬁcations with

bag-of-words features that are based on an abusive ex-

pression dictionary.

In this paper, we ﬁrst present a rather new task of

detecting and collecting documents that contain heart-

warming stories from a huge variety of contents on the

Web. More speciﬁcally, since we target tweets on the

social networking site Twitter(https://twitter.com/),

our task is to detect tweets that contain heartwarm-

ing stories from the huge amount of tweets posted

on Twitter. We then present a method for identifying

heartwarming tweets. We believe the task for detect-

ing documents (tweets) containing heartwarming sto-

ries is new in that they should not only make readers

happy (have positive sentiment from the reader’s per-

spective) but contain the writer’s actual experiences.

323

Okumura M., Yamaguchi Y., Suzuki M. and Otsuka H..

Identifying Tweets that Contain a ’Heartwarming Story’.

DOI: 10.5220/0005129503230326

In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval (KDIR-2014), pages 323-326

ISBN: 978-989-758-048-2

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

2 TWEETS THAT CONTAIN

HEARTWARMING STORIES

There have been many sites on the Web that are dedi-

cated to heartwarming stories

. Those stories on these

sites tend to touch the readers’ hearts. The visitors of

the sites will ﬁnd a lot of joy in these stories. Heart-

warming stories are said to be the stories that will

make readers laugh, cry, and smile. However, the sto-

ries on these sites are manually collected.

Our task is to detect tweets that contain such heart-

warming stories from a huge amount of tweets on

Twitter. The following is a sample tweet that is con-

sidered heartwarming

6B.M®o/ÆùL

*(S3+8ÎVS.J*

#&I#C>.K+#(

7

(I was really tired. I stopped at the conve-

nience store because I could not make a boxed

lunch today. I was woken up by the staff’s ca-

sual words of, ’Have a nice day.’)

6ÏÚ(@#&¯>#).

M'&SGS×#&I>

$& #VSª7

(I thought I missed the last train when I fell

down just in front of the closing door but the

st aff kindly opened the door again. He was

kind.)

3 COLLECTING

HEARTWARMING TWEETS

We tried to construct a corpus of heartwarming

tweets. Since it is difﬁcult to collect heartwarming

tweets from the vast number of topics on Twitter,

we decided to construct a corpus of heartwarming

tweets that contain episodes concerning railway and

convenience-store staffs. We chose these two groups

of individuals because we often come in daily con-

tact with them as customers. We selected two railway

companies and a convenience store chain in Japan.

We ﬁrst retrieved a collection of tweets from Twit-

ter that contained the following query: Company

name + Keywords indicating company staffs

. We

then tried to automatically remove the following types

of tweets:

For example, http://www.heartwarmingstories.net/.

We manually rephrased original tweets more formally.

For example, 6;7 (conductor), 6V7 (station

staff), 6ÎV7(store staff), and so on.

• tweets created by bots,

• retweets, and

• tweets considered news articles.

Next, we manually judged whether each tweet

contained a heartwarming story by actually reading it.

The statistics of the corpus are shown in Table 1. As

seen in this table, the ratio of heartwarming tweets is

quite small, and therefore, it is challenging to identify

them from a complete collection of tweets.

4 DETECTING HEARTWARMING

TWEETS

Our method for detecting heartwarming tweets from a

collection of tweets is based on a classiﬁcation model

trained by using SVM. We used SVM because the

learning algorithms have been successfully used in

text classiﬁcation in the past.

The simplest way to construct a model for detect-

ing heartwarming tweets using supervised learning al-

gorithms is to use a ’bag of words’ (BoW) whose

parts-of-speech are a verb, a noun, and a sufﬁx in the

tweet as the features. We used sufﬁxes because ben-

eﬁciary expressions such as 6&LK7 and 6&

BI7 are sufﬁxes.

However, we took into account the following ideas

and devised better features to improve the predic-

tion accuracy, because heartwarming tweets should

not only contain positive sentiments from the reader’s

perspective but also contain the actual experiences of

the writer:

1. When writers describe their positive feelings in a

tweet, readers might empathize and thus become

happy. Therefore, we used the lexica of polarity-

bearing words and calculate the number of posi-

tive and negative words as features.

We used two sentiment lexica as our lexical re-

source for the polarity-bearing words. The ﬁrst

is available on the Web

. The lexicon was con-

structed using the method developed by Takamura

et al.(Takamura et al., 2005). The second was con-

structed using the method developed by Suzuki et

al.(Suzuki et al., 2007).

2. Since heartwarming tweets should also contain

actual experiences, clues for whether a tweet con-

tains a story should be incorporated. Therefore,

we used the length of the tweet and whether or

not the tweet contained verbs in the past tense as

features

http://www.lr.pi.titech.ac.jp/∼takamura/pndic en.html

Tweets describing a story might be long.

KDIR2014-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval

324

Table 1: Statistics of heartwarming tweets corpus.

Company Total No. Heartwarming Non-heartwarming

of tweets tweets tweets

Family-Mart 691 71 620

JR 657 48 609

Tokyo Metro 664 74 590

3. As noted in Table 1, our data set is severely im-

balanced and the heartwarming tweets class has

fewer examples than the other. Learning algo-

rithms that do not take into account the class im-

balance tend to be overwhelmed by the majority

class and ignore the minority one, and the over-

all accuracy of conventional learning algorithms

will thus signiﬁcantly degrade, since their classi-

ﬁers are greatly biased towards the majority class.

Therefore, under-sampling (Zhang et al., 2010) is

used and the training is performed using an equal

number of positive and negative examples.

The sampling came from a class of methods that

alters the size of the training sets. Under-sampling

is an imbalanced data learning method that uses

only a subset of the majority class examples.

Under-sampling changes the training sets by ran-

domly sampling the examples from the major-

ity class training set and making it smaller. The

level of imbalance is reduced, with the hope that a

more balanced training set can provide better re-

sults. Among the various class-imbalance learn-

ing methods, under-sampling has been commonly

used. The under-sampling method we adopted

clusters the examples in the majority class ﬁrst

and then randomly selects examples from these

clusters.

5 EXPERIMENTS

In this section, we report on the experimental results

using our prediction method described in the previous

section on our data collection described in Section 3.

We used the standard precision and recall for the

positive class of heartwarming tweets, and the F1

measure, which is the harmonic mean between the

precision and recall, to evaluate the prediction. We

used the libsvm implementation (Fan et al., 2005)

for SVM with the linear kernel in the 10-fold cross-

validation.

We compared the following different feature sets

for the prediction:

1. BoW whose parts-of-speech are a verb, a noun,

and a sufﬁx in the tweet (baseline feature set),

Table 2: Prediction performance.

Family-Mart:

Feature Precision Recall F1-measure

Baseline 0.534 0.775 0.632

+ polarity 0.518 0.817 0.634

+ tense 0.529 0.775 0.629

+ length 0.417 0.845 0.558

All 0.378 0.789 0.511

JR:

Feature Precision Recall F1-measure

Baseline 0.183 0.625 0.283

+ polarity 0.208 0.729 0.324

+ tense 0.172 0.729 0.278

+ length 0.163 0.792 0.270

All 0.284 0.646 0.395

Tokyo Metro:

Feature Precision Recall F1-measure

Baseline 0.384 0.757 0.510

+ polarity 0.341 0.757 0.471

+ tense 0.364 0.797 0.500

+ length 0.290 0.784 0.423

All 0.425 0.838 0.564

2. BoW + polarity information (Please see 1. in Sec-

tion 4),

3. BoW + past tense (Please see 2. in Section 4),

4. BoW + tweet length (Please see 2. in Section 4),

5. BoW + polarity + past tense + tweet length + other

features (all features).

’Other features’ in the above feature sets indicate

whether interjections or quotation marks are in the

tweet. The experiments were performed using under-

sampling for all the feature sets.

In Table 2, we show the prediction results for the

three companies mentioned in Section 3 using our

method. The best performance is shown in boldface

for each company.

We found by comparing the feature sets that

adding sentiment features mostly improves the per-

formance. However, simply adding the features for

detecting a story in a tweet (past tense and tweet

length) cannot contribute to improving the perfor-

mance, while adding all the features to the baseline

feature set mostly yields the best performance from

among the feature sets.

IdentifyingTweetsthatContaina'HeartwarmingStory'

325

6 CONCLUSIONS

In this paper, we ﬁrst presented a rather new task

of detecting and collecting tweets that contain heart-

warming stories from a huge amount of tweets on

Twitter. We then presented a method for identify-

ing heartwarming tweets. Our prediction method is

based on a supervised learning algorithm in SVM

along with the features from the tweets.

We need to improve the prediction performance

by devising more intelligent features for future work.

Furthermore, we also need to construct larger corpus

of heartwarming tweets for the experiments. Trying

the task in other languages such as English is also our

future work.

ACKNOWLEDGEMENTS

This work is conducted under a joint project with the

Survey Research Center Co. Ltd. and The Insti-

tute of Behavioral Sciences in Japan. We would like

to thank the following co-workers for their valuable

comments and suggestions: Hiroshi Tomatsuri, Kiy-

ohisa Takano, Hiroharu Ochihara, Yuichi Mohri, and

Hiromi Ito.

REFERENCES

Fan, R., Chen, P., and Lin, C. (2005). Working set selection

using second order information for training support

vector machines. The Journal of Machine Learning

Research.

Hasegawa, T., Kaji, N., and Yoshinaga, N. (2013). Pre-

dicting and eliciting addressee’s emotion in online di-

alogue. In Proceedings of the 51st Annual Meet-

ing of the Association for Computational Linguistics

(ACL2013), pages 964–972.

Lin, K. H.-Y., Yang, C., and Chen, H.-H. (2008). Emotion

classiﬁcation of online news articles from the reader’s

perspective. In Proc. of the 2008 IEEE/WIC/ACM In-

ternational Conference on Web Intelligence (WI’08).

Liu, B. (2012). Sentiment Analysis and Opinion Mining.

Morgan & Claypool Publishers.

Pang, B. and Lee, L. (2008). Opinion mining and sentiment

analysis. Foundations and Trends in Information Re-

trieval, 2(1-2):1–135.

Razavi, A. H., Inkpen, D., Uritsky, S., and Matwin, S.

(2010). Offensive language detection using multi-

level classiﬁcation. In Proceedings of the 23rd Cana-

dian conference on Advances in Artiﬁcial Intelligence

(AI’10), pages 16–27.

Suzuki, Y., Takamura, H., and Okumura, M. (2007). Semi-

supervised learning to classify evaluative expressions

from labeled and unlabeled texts. IEICE Transactions

on Information and Systems, pages 1516–1522.

Takamura, H., Inui, T., and Okumura, M. (2005). Extract-

ing semantic orientations of words using spin model.

In Proceedings of the 43rd Annual Meeting of the As-

sociation for Computational Linguistics (ACL2005),

pages 133–140.

Yang, C., Lin, K. H.-Y., and Chen, H.-H. (2009). Writer

meets reader: Emotion analysis of social media from

both the writer’s and reader’s perspectives. In Proc.

of the 2009 IEEE/WIC/ACM International Conference

on Web Intelligence (WI’09), pages 287–290.

Zhang, Y., Zhang, L., and Wang, Y. (2010). Cluster-based

majority under-sampling approaches for class imbal-

ance learning. In Proceedings of the 2nd IEEE In-

ternational Conference on Information and Financial

Engineering (ICIFE), pages 400–404.

KDIR2014-InternationalConferenceonKnowledgeDiscoveryandInformationRetrieval

326