An Empirical Study of the Effectiveness of using Sentiment Analysis Tools for Opinion Mining

Tao Ding, Shimei Pan

2016

Abstract

Sentiment analysis is increasingly used as a tool to gauge people’s opinions on the internet. For example, sentiment analysis has been widely used in assessing people’s opinions on hotels, products (e.g., books and consumer electronics), public policies, and political candidates. However, due to the complexity in automated text analysis, today’s sentiment analysis tools are far from perfect. For example, many of them are good at detecting useful mood signals but inadequate in tracking and inferencing the relationships between different moods and different targets. As a result, if not used carefully, the results from sentiment analysis can be meaningless or even misleading. In this paper, we present an empirical analysis of the effectiveness of using existing sentiment analysis tools in assessing people’s opinions in five different domains. We also proposed several effectiveness indicators that can be computed automatically to help avoid the potential pitfalls in misusing a sentiment analysis tool.

References

  1. Andreevskaia, A. and Bergler, S. (2006). Unsupervised sentiment analysis with emotional signals. In 11rd Conference of the European Chapter of the Association for Computational Linguistics, EACL-06.
  2. Belkin, D. (2015). Harvard accused of bias against asianamericans. The Wall Street Journal.
  3. Bob Lannon, A. P. (2014). What can we learn from 800,000 public comments on the fcc's net neutrality plan? @ONLINE.
  4. Chung, W. and Zeng, D. (2015). Social-media-based public policy informatics: Sentiment and network analyses of u.s. immigration and border security. Journal of the Association for Information Science and Technology, pages n/a-n/a.
  5. Diakopoulos, N. A. and Shamma, D. A. (2010). Characterizing debate performance via aggregated twitter sentiment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 7810, pages 1195-1198, New York, NY, USA. ACM.
  6. Ghose, A., Ipeirotis, P. G., and Sundararajan, A. (2007). Opinion mining using econometrics: A case study on reputation systems. In In Proceedings of the 44th Annual Meeting of the Association for Computational Linguistics (ACL.
  7. Hu, X., Tang, J., Gao, H., and Liu, H. (2013). Unsupervised sentiment analysis with emotional signals. In Proceedings of the 22Nd International Conference on World Wide Web, WWW 7813, pages 607-618, Republic and Canton of Geneva, Switzerland. International World Wide Web Conferences Steering Committee.
  8. Kilgarriff, A. and Rose, T. (1998). Measures for corpus similarity and homogeneity. In 3rd Conf. on Empirical Methods in Natural Language Processing.
  9. Liu, B. (2012). Sentiment Analysis and Opinion Mining. Morgan and Claypool.
  10. Maynard, D. and Funk, A. (2012). Automatic detection of political opinions in tweets. In Proceedings of the 8th International Conference on The Semantic Web, ESWC'11, pages 88-99, Berlin, Heidelberg. Springer-Verlag.
  11. Pang, B. and Lee, L. (2004a). A sentimental education: Sentiment analysis using subjectivity. In Proceedings of ACL, pages 271-278.
  12. Pang, B. and Lee, L. (2004b). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42Nd Annual Meeting on Association for Computational Linguistics, ACL 7804, Stroudsburg, PA, USA. Association for Computational Linguistics.
  13. Pang, B. and Lee, L. (2008). Opinion mining and sentiment analysis. Found. Trends Inf. Retr., 2(1-2):1-135.
  14. Pang, B., Lee, L., and Vaithyanathan, S. (2002). Thumbs up?: Sentiment classification using machine learning techniques. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing - Volume 10, EMNLP 7802, pages 79-86, Stroudsburg, PA, USA. Association for Computational Linguistics.
  15. Pappas, N., Katsimpras, G., and Stamatatos, E. Distinguishing the popularity between topics: A system for up-todate opinion retrieval and mining in the web. In 14th International Conference on Intelligent Text Processing and Computational Linguistics.
  16. Plackett, R. L. (1983). Karl pearson and the chi-squared test. International Statistical Review, 51:59-72.
  17. Read, J. and Carroll, J. (2009). Weakly supervised techniques for domain-independent sentiment classification. In Proceedings of the 1st International CIKM Workshop on Topic-sentiment Analysis for Mass Opinion, TSA 7809, pages 45-52, New York, NY, USA. ACM.
  18. Shi, H.-X. and Li, X.-J. (2011). A sentiment analysis model for hotel reviews based on supervised learning. In Machine Learning and Cybernetics (ICMLC), 2011 International Conference on, volume 3, pages 950-954.
  19. Singhal, A. (2001). Modern information retrieval: A brief overview. IEEE Data Eng. Bull., 24(4):35-43.
  20. Turney, P. D. (2002). Thumbs up or thumbs down?: Semantic orientation applied to unsupervised classification of reviews. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL 7802, pages 417-424, Stroudsburg, PA, USA. Association for Computational Linguistics.
  21. Wang, H., Can, D., Kazemzadeh, A., Bar, F., and Narayanan, S. (2012). A system for real-time twitter sentiment analysis of 2012 u.s. presidential election cycle. In Proceedings of the ACL 2012 System Demonstrations, ACL 7812, pages 115-120, Stroudsburg, PA, USA. Association for Computational Linguistics.
  22. Wang, H., Lu, Y., and Zhai, C. (2011). Latent aspect rating analysis without aspect keyword supervision. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 7811, pages 618-626, New York, NY, USA. ACM.
  23. Wei Peng, D. H. P. (2011). Generate adjective sentiment dictionary for social media sentiment analysis using constrained nonnegative matrix factorization. In The International AAAI Conference on Web and Social Media, ICWSM.
  24. Zhou, X., Tao, X., Yong, J., and Yang, Z. (2013). Sentiment analysis on tweets for social events. In Computer Supported Cooperative Work in Design (CSCWD), 2013 IEEE 17th International Conference on, pages 557- 562.
Download


Paper Citation


in Harvard Style

Ding T. and Pan S. (2016). An Empirical Study of the Effectiveness of using Sentiment Analysis Tools for Opinion Mining . In Proceedings of the 12th International Conference on Web Information Systems and Technologies - Volume 2: WEBIST, ISBN 978-989-758-186-1, pages 53-62. DOI: 10.5220/0005760000530062


in Bibtex Style

@conference{webist16,
author={Tao Ding and Shimei Pan},
title={An Empirical Study of the Effectiveness of using Sentiment Analysis Tools for Opinion Mining},
booktitle={Proceedings of the 12th International Conference on Web Information Systems and Technologies - Volume 2: WEBIST,},
year={2016},
pages={53-62},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005760000530062},
isbn={978-989-758-186-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Conference on Web Information Systems and Technologies - Volume 2: WEBIST,
TI - An Empirical Study of the Effectiveness of using Sentiment Analysis Tools for Opinion Mining
SN - 978-989-758-186-1
AU - Ding T.
AU - Pan S.
PY - 2016
SP - 53
EP - 62
DO - 10.5220/0005760000530062