Automatic Illustration of Short Texts via Web Images

Sandro Aldo Aramini, Edoardo Ardizzone, Giuseppe Mazzola

2015

Abstract

In this paper we propose a totally unsupervised and automatic illustration method, which aims to find onto the Web a set of images to illustrate the content of an input short text. The text is modelled as a semantic space and a set of relevant keywords is extracted. We compare and discuss different methods to create semantic representations by keyword extraction. Keywords are used to query Google Image Search engine for a list of relevant images. We also extract information from the Web pages that include the retrieved images, to create an Image Semantic Space, which is compared to the Text Semantic Space in order to rank the list of retrieved images. Tests showed that our method achieves very good results, which overcome those obtained by using a state-of-the-art application. Furthermore we developed a Web tool to test our system and evaluate results within the Internet community.

References

  1. Barnard, K., Duygulu, P., et al., 2003. Matching words and pictures. JMLR, 3:1107-1135.
  2. Barnard, K., and Forsyth, D., 2001. Learning the Semantics of Words and Pictures. Proc. International Conference on Computer Vision, pp. II: 408-415, 2001.
  3. Carney, R. N., and Levin, J. R., 2002, “Pictorial illustrations still improve students' learning from text”, Educational Psychology Review, 2002, 14(1), 5-26.
  4. Carneiro, G., Chan, A., et al., 2007. Supervised learning of semantic classes for image annotation and retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3):394-410.
  5. Carney, R. N., and Levin, J. R., 2002. Pictorial illustrations still improve students' learning from text. Educational Psychology Review, 14(1), 5-26.
  6. Coelho, F., and Ribeiro, C., 2011, Automatic illustration with cross-media retrieval in large-scale collections. In Content-Based Multimedia Indexing (CBMI), 2011 9th International Workshop on (pp. 25-30). IEEE.
  7. Coyne, B., and Sproat, R., 2001. Wordseye: An automatic text-to-scene conversion system. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. SIGGRAPH 2001, 487-496.
  8. Deerwester, S., Dumais, S. T., et al., 1990. Indexing by latent semantic analysis. Journal of the American Society For Information Science, 41(6), 391-407.
  9. Delgado, D., Magalhaes, J., & Correia, N., 2010. Automated illustration of news stories. In Semantic Computing (ICSC), 2010 IEEE Fourth International Conference on (pp. 73-78). IEEE.
  10. Feng, Y., and Lapata, M., 2010. Topic models for image annotation and text illustration. In Proceedings of the NAACL HLT. Association for Computational Linguistics, Los Angeles, California, pages 831-839.
  11. Feng, S., Manmatha, R, and Lavrenko, V., 2004. Multiple bernoulli relevance models for image and video annotation. In CVPR, volume 2(2004), pp. 1002-1009.
  12. Joshi, D., Wang, J .Z., and Li, J., 2006. The story picturing engine-a system for automatic text illustration. ACM Transactions on Mul-timedia Computing, Communications, and Applications, 2(1):68-89.
  13. Kandola, J. S., Shawe-Taylor, J., and Cristianini, N., 2003. Learning semantic similarity, In Neural Information Processing Systems 15 (NIPS 15), pp. 657-664.
  14. Lowe, W., 2001. Towards a theory of semantic space. In Proceedings of the Twenty-Third Annual Conference of the Cognitive Science Society 2001 (pp. 576-581). Mahwah, NJ: Erlbaum.
  15. Miller, G. 1990. WordNet: An on-line lexical database. Int. Journal of Lexicography, Special Issue, 3(4).
  16. Monay, F., and Gatica-Perez, D., 2007. Modeling semantic aspects for cross-media image indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(10):1802-1817.
  17. Rasiwasia, N., Pereira, J. C., et al. 2010. A new approach to cross-modal multimedia retrieval. In Proceedings of the International Conference on Multimedia (MM 7810), 251-260.
  18. Yutaka, M., and Ishizuka, M., 2004. Keyword extraction from a single document using word co-occurrence statistical information. Int'l Journal on Artificial Intelligence Tools, 13(1):157-169.
  19. Zhu, X., Goldberg, A. B., et al., 2007. A text-to-picture synthesis system for augmenting communication. In Proceedings of the 22nd national conference on Artificial intelligence, Vol. 2, 1590-1595 2007.
  20. link1: http://en.wikinews.org/wiki/Main_Page.
  21. link2: http://alipr.com/spe/
  22. link3: http://en.wikinews.org/wiki/Los_Angeles_Lakers_ need_to_win_game_six_to_tie_NBA_championship.
Download


Paper Citation


in Harvard Style

Aramini S., Ardizzone E. and Mazzola G. (2015). Automatic Illustration of Short Texts via Web Images . In Proceedings of the 6th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2015) ISBN 978-989-758-088-8, pages 139-148. DOI: 10.5220/0005307301390148


in Bibtex Style

@conference{ivapp15,
author={Sandro Aldo Aramini and Edoardo Ardizzone and Giuseppe Mazzola},
title={Automatic Illustration of Short Texts via Web Images},
booktitle={Proceedings of the 6th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2015)},
year={2015},
pages={139-148},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005307301390148},
isbn={978-989-758-088-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2015)
TI - Automatic Illustration of Short Texts via Web Images
SN - 978-989-758-088-8
AU - Aramini S.
AU - Ardizzone E.
AU - Mazzola G.
PY - 2015
SP - 139
EP - 148
DO - 10.5220/0005307301390148