Ordering Matters: An Experimental Study of Ranking Influence on Results Selection Behavior during Exploratory Search

Emanuel Felipe Duarte, Lucas Pupulin Nanni, Ricardo Theis Geraldi, Edson OliveiraJr, Valéria Delisandra Feltrim, Roberto Pereira

2016

Abstract

The design of Exploratory Search tools acquires more importance as the amount of information on the Web grows. Accordingly, informed design decisions concerning the users’ behavior during search activities can be used in novel approaches for Exploratory Search tools. With regard to users’ behavior on search result selection, literature indicates higher ranked results tend to attract more attention and, therefore, more hits. However, the cause of such behavior is not clear. We experimentally investigate the hypothesis of this behavior being due to ranking. A group of 72 participants was asked to select a search result from a randomly ordered results list. The experiment was carried out on paper to remove specific search engine and digital medium biases. We obtained evidence indicating there is a trend towards choosing higher ranked results even in a different medium and in the context of an Exploratory Search, corroborating to the hypothesis that ranking has influence on users’ selection behavior.

References

  1. Aula, A., Majaranta, P., and Räihä, K.-J. (2005). Eyetracking reveals the personal styles for search result evaluation. In Proc. Int. Conf. IFIP TC13 on HumanComputer Interaction, pages 1058-1061. SpringerVerlag.
  2. Basili, V., Caldiera, G., and Rombach, H. D. (1994). Encyclopedia of Software Engineering, chapter The Goal Question Metric Approach, pages 528-532. John Wiley & Sons, Inc.
  3. Bates, M. J. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Information Review, 13(5):407-424.
  4. Blandford, A. and Attfield, S. (2010). Interacting with Information, volume 3 of Synthesis Lectures on HumanCentered Informatics. Morgan & Claypool Publishers.
  5. Eisenberg, M. and Barry, C. (1988). Order effects: A study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39(5):293- 300.
  6. Granka, L. A., Joachims, T., and Gay, G. (2004). Eyetracking analysis of user behavior in www search. In Proc. Int. Conf. ACM SIGIR on Research and Development in Information Retrieval, pages 478-479. ACM.
  7. Hearst, M. (2009). Search User Interfaces. Search User Interfaces. Cambridge University Press.
  8. Jansen, B. J., Zhang, L., and Mattila, A. S. (2012). User reactions to search engines logos: investigating brand knowledge of web search engines. Electronic Commerce Research, 12(4):429-454.
  9. Jansen, B. J., Zhang, M., and Schultz, C. D. (2009). Brand and its effect on user perception of search engine performance. Journal of the American Society for Information Science and Technology, 60(8):1572-1595.
  10. Joachims, T., Granka, L., Pan, B., Hembrooke, H., and Gay, G. (2005). Accurately interpreting clickthrough data as implicit feedback. In Proc. Int. Conf. ACM SIGIR on Research and Development in Information Retrieval, pages 154-161. ACM.
  11. Kules, B. and Capra, R. (2009). Designing exploratory search tasks for user studies of information seeking support systems. In Proc. ACM/IEEE-CS Joint Conf. on Digital libraries, pages 419-420.
  12. Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., and Granka, L. (2007). In google we trust: Users decisions on rank, position, and relevance. Journal of Computer-Mediated Communication, 12(3):801-823.
  13. Russell, D. and Grimes, C. (2007). Assigned tasks are not the same as self-chosen web search tasks. In Proc. Int. Conf. HICSS on System Sciences, pages 83-83.
  14. Van Solingen, R. and Berghout, E. (1999). The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development. McGraw-Hill Higher Education.
  15. Weinschenk, S. (2011). 100 Things Every Designer Needs to Know About People. New Riders Publishing.
  16. White, R. W., Capra, R., Golovchinsky, G., Kules, B., Smith, C., and Tunkelang, D. (2013). Introduction to special issue on humancomputer information retrieval. Information Processing & Management, 49(5):1053 - 1057.
  17. White, R. W., Kules, B., and Bederson, B. (2005). Exploratory search interfaces: categorization, clustering and beyond: report on the XSI workshop at the human-computer interaction laboratory, university of maryland. SIGIR Forum, 39(2):52-56.
  18. White, R. W. and Roth, R. A. (2009). Exploratory Search: Beyond the Query-Response Paradigm, volume 1 of Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool Publishers.
  19. Wildemuth, B. M. and Freund, L. (2012). Assigning search tasks designed to elicit exploratory search behaviors. In Proc. Symp. on Human-Computer Interaction and Information Retrieval, pages 4:1-4:10. ACM.
  20. Wohlin, C., Höst, M., Runeson, P., Ohlsson, M., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering. Computer Science. Springer.
Download


Paper Citation


in Harvard Style

Duarte E., Nanni L., Geraldi R., OliveiraJr E., Feltrim V. and Pereira R. (2016). Ordering Matters: An Experimental Study of Ranking Influence on Results Selection Behavior during Exploratory Search . In Proceedings of the 18th International Conference on Enterprise Information Systems - Volume 1: ICEIS, ISBN 978-989-758-187-8, pages 427-434. DOI: 10.5220/0005828704270434


in Bibtex Style

@conference{iceis16,
author={Emanuel Felipe Duarte and Lucas Pupulin Nanni and Ricardo Theis Geraldi and Edson OliveiraJr and Valéria Delisandra Feltrim and Roberto Pereira},
title={Ordering Matters: An Experimental Study of Ranking Influence on Results Selection Behavior during Exploratory Search},
booktitle={Proceedings of the 18th International Conference on Enterprise Information Systems - Volume 1: ICEIS,},
year={2016},
pages={427-434},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005828704270434},
isbn={978-989-758-187-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 18th International Conference on Enterprise Information Systems - Volume 1: ICEIS,
TI - Ordering Matters: An Experimental Study of Ranking Influence on Results Selection Behavior during Exploratory Search
SN - 978-989-758-187-8
AU - Duarte E.
AU - Nanni L.
AU - Geraldi R.
AU - OliveiraJr E.
AU - Feltrim V.
AU - Pereira R.
PY - 2016
SP - 427
EP - 434
DO - 10.5220/0005828704270434