Performance Evaluation of State-of-the-Art Ranked Retrieval Methods and Their Combinations for Query Suggestion

Suthira Plansangket, John Q. Gan

2014

Abstract

This paper investigates several state-of-the-art ranked retrieval methods, adapts and combines them as well for query suggestion. Four performance criteria plus user evaluation have been adopted to evaluate these query suggestion methods in terms of ranking and relevance from different perspectives. Extensive experiments have been conducted using carefully designed eighty test queries which are related to eight topics. The experimental results show that the method developed in this paper, which combines the TF-IDF and Jaccard coefficient methods, is the best method for query suggestion among the six methods evaluated, outperforming the most popularly used TF-IDF method. Furthermore, it is shown that re-ranking query suggestions using Cosine similarity improves the performance of query suggestions.

References

  1. Baeza-Yates, R. and Ribeiro-Neto, B., 2011. Modern information retrieval: the concepts and technology behind search. England: Pearson Education Limited.
  2. Baeza-Yates, R., Hurtado, C., and Mendoza, M., 2004. Query recommendation using query logs in search engines. In Proc. of the 2004 International Conference on Current Trends in Database Technology, page 588- 596
  3. Biega, J., Kuzey, E., and Suchanek, F., 2013. Inside YAGO2s: A transparent information extraction architecture. In Proc. of WWW 2013, Rio de Janeiro, Brazil.
  4. Blizard, W. D., 1989. Multiset theory. Notre Dame Journal of Formal Logic, vol. 30, no. 1, page 36-66.
  5. Boldi, P., Bonchi, F., Castillo, C., Donato, D., and Vigna, S., 2009. Query suggestion using query flow graphs. In Proc. of the 2009 Workshop on Web Search Click Data, Milan, Italy, page 56-63.
  6. Boldi, P., Bonchi, F., Castillo, C., Donato, D., Gionis, A, and Vigna, S., 2008. The query flow graph: model and applications. In Proc. of CIKM'08, California, USA.
  7. Bordag, S., 2008. A comparison of co-occurrence and similarity measures as simulations of context. In Proc. of the 9th International Conference on Computational Linguistics and Intelligent Text Processing, SpringerVerlag Berlin, Heidelberg, page 52-63.
  8. Cao, H., Jiang, D., Pei, J., He, Q., Liao, Z., Chen, E., and Li, H., 2008. Context-aware query suggestion by mining click-through and session data. In Proc. of KDD'2008, Nevada, USA.
  9. Costa, M., Miranda, J., Cruz, D., and Gomes, D., 2012. Query suggestion for web archive search. In Proc. of the 10th International Conference on Preservation of Digital Objects (iPres 2013), Lisbon, Portugal.
  10. Delgado, M., Martin-Bautista, M.J., Sanchez, D., Serrano, J.M., and Vila, M.A., 2009. Association rules and fuzzy association rules to find new query terms. In Proc. of the Third Conference of the EUSFLAT, Lisbon, Portugal, page 49-53.
  11. Dybkjaer, L., Hemsen, H., and Minker, W. (Eds.), 2007. Evaluation of text and speech systems. Springer, Dordrecht, Netherlands.
  12. Fonseca, B.M., Golgher, P.B., de Moura, E.S., and Ziviani N., 2003. Using association rules to discover search engines related queries. In Proc. of The First Latin American Web Congress, USA, page 66-71.
  13. Gong, Z., Cheang, C., and Hou, L., 2005. Web query expansion by WordNet. In LNCS 3588, page 166-175.
  14. He, B. and Ounis, I., 2009. Studying query expansion effectiveness. In Proc. of the 31th European Conference on IR Research on Advances in Information Retrieval, Toulouse, France , page 611 - 619.
  15. Hoffart, J., Suchanek, F., Berberich, K., Lewis-Kelham, E., Melo, G., and Weikum, G., 2011. YAGO2: exploring and querying world knowledge in time, space, context, and many languages. In Proc. of WWW 2011, Hyderabad, India.
  16. Hu, H., Zhang, M., He, Z., Wang, P., and Wang, W., 2013. Diversifying query suggestions by using topics from Wikipedia. In Proc. of the 2013 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, Atlanta, GA.
  17. Huang, C., Chien, L., and Oyang, Y., 2003. Relevant term suggestion in interactive web search based on contextual information in query session logs. Journal of the American Society for Information \Science and Technology, vol. 54, no. 7, page 638-649.
  18. Jurafsky, D. and Martin, J. H., 2008. Speech and language processing: an introduction to natural language processing. Computational Linguistics and Speech Recognition. Second Edition, Prentice Hall.
  19. Kato, M., Sakai, T., and Tanaka, K. 2011. Query session data vs. clickthrough data as query suggestion resources. In Proc. of ECIR 2011, Dublin, Ireland.
  20. Kato, M., Sakai, T., and Tanaka, K., 2012. Structured query suggestion for specialization and parallel movement: effect on search behaviors. In Proc. of WWW 2012, Lyon, France, page 389-398.
  21. Kato, M., Sakai, T., and Tanaka, K., 2013. When do people use query suggestion? A query suggestion log analysis. Information Retrieval, vol. 16, no. 6, page 725-746.
  22. Kelly, D., Cushing, A., Dostert, M., Niu, X., and Gyllstrom, K., 2010. Effects of popularity and quality on the usage of query suggestions during information search. In Proc. of CHI'2010, Atlanta, USA.
  23. Kruschwitz, U., Lungley, D., Albakour, M. and Song, D., 2013. Deriving query suggestions for site search. In Journal of the American Society for Information Science and Technology, vol. 64, no. 10, page 1975- 1994.
  24. Kulkarni, S. and Caragea, D., 2009. Computation of the semantic relatedness between words using concept clouds. In Proc. of KDIR 2009, page 183-188.
  25. Liao, Z., Song, Y., Huang, Y., He, L., and He, Q. 2014. Task trail: an effective segmentation of user search behavior. IEEE Transactions on Knowledge and Data Engineering (in press)
  26. Manning, C. D., Raghavan, P., and Schutze, H., 2008. Introduction to information retrieval. England: Cambridge University Press.
  27. Mei, Q., Zhou, D., and Church, K., 2008. Query suggestion using hitting time. In Proc. of CIKM'08, California, USA.
  28. Nallapati, R. and Shah, C., 2006. Evaluating the quality of query refinement suggestions in information retrieval. In Proc. of CIKM 2006, Arlington, Virginia, USA.
  29. Okabe, M. and Yamada, S., 2007. Semisupervised query expansion with minimal feedback. IEEE Transactions on Knowledge and Data Engineering, vol. 19, no. 11, page 1585-1589.
  30. Otegi, A., Arregi, X., and Agirre, E., 2011. Query expansion for IR using knowledge-based relatedness. In Proc. of the 5th International Joint Conference on NLP, Chang Mai, Thailand, page 1467-1471.
  31. Song, Y., Zhou, D., and He, L., 2012. Query suggestion by constructing term-transition graphs. In Proc. of WSDM'12, Seattle, Washington, USA.
  32. Suchanek, F., Kasneci, G., and Weikum, G., 2007. YAGO: a core of semantic knowledge unifying WordNet and Wikipedia. In Proc. of WWW 2007, Banff, Alberta, Canada.
  33. Suchanek, F., Hoffart, J., Kuzey, E., Lewis-Kelham, E., 2013. YAGO2s: modular high-quality information extraction with an application to flight planning. In Proc. of the German Computer Science Symposium (BTW 2013), Magdeburg, Germany.
  34. Wan, J., Wang, W., Yi, J., Chu, C., and Song, K., 2012. Query expansion approach based on ontology and local context analysis. Research Journal of Applied Sciences, Engineering and Technology, vol. 4, no. 16, page 2839-2843.
  35. Yang, J., Cai, R., Jing, F., Wang, S., Zhang, L., and Ma, W., 2008. Search-based query suggestion. In Proc.of CIKM'08, California, USA.
  36. Yih, W. and Qazvinian, V., 2012. Measuring word relatedness using heterogeneous vector space models. In Proc. of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT-2012), Montreal, Canada.
  37. Zanon, R., Albertini, S., Carullo, M., and Gallo, I., 2012. A new query suggestion algorithm for taxonomybased search engines. In Proc. of KDIR 2012, page 151-156.
Download


Paper Citation


in Harvard Style

Plansangket S. and Q. Gan J. (2014). Performance Evaluation of State-of-the-Art Ranked Retrieval Methods and Their Combinations for Query Suggestion . In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2014) ISBN 978-989-758-048-2, pages 141-148. DOI: 10.5220/0005018401410148


in Bibtex Style

@conference{kdir14,
author={Suthira Plansangket and John Q. Gan},
title={Performance Evaluation of State-of-the-Art Ranked Retrieval Methods and Their Combinations for Query Suggestion},
booktitle={Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2014)},
year={2014},
pages={141-148},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005018401410148},
isbn={978-989-758-048-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2014)
TI - Performance Evaluation of State-of-the-Art Ranked Retrieval Methods and Their Combinations for Query Suggestion
SN - 978-989-758-048-2
AU - Plansangket S.
AU - Q. Gan J.
PY - 2014
SP - 141
EP - 148
DO - 10.5220/0005018401410148