![](bgc.png)
Araujo, T. (2020). Conversational agent research toolkit:
An alternative for creating and managing chatbots for
experimental research. Computational Communica-
tion Research, 2(1):35–51.
Avula, S. and Arguello, J. (2020). Wizard of oz interface
to study system initiative for conversational search. In
Proceedings of the 2020 Conference on Human Infor-
mation Interaction and Retrieval, pages 447–451.
Avula, S., Arguello, J., Capra, R., Dodson, J., Huang,
Y., and Radlinski, F. (2019). Embedding search
into a conversational platform to support collabora-
tive search. In Proceedings of the 2019 Conference on
Human Information Interaction and Retrieval, pages
15–23.
Avula, S., Chadwick, G., Arguello, J., and Capra, R.
(2018). Searchbots: User engagement with chat-
bots during collaborative search. In Proceedings of
the 2018 Conference on Human Information Interac-
tion&Retrieval, pages 52–61. ACM.
Bailey, P., Moffat, A., Scholer, F., and Thomas, P. (2016).
UQV100: A test collection with query variability.
In Proceedings of the 39th International ACM SIGIR
Conference on Research and Development in Informa-
tion Retrieval, SIGIR ’16, pages 725–728, New York,
NY, USA. ACM.
Bowden, K. K., Oraby, S., Wu, J., Misra, A., and Walker,
M. (2017). Combining search with structured data to
create a more engaging user experience in open do-
main dialogue. arXiv preprint arXiv:1709.05411.
Bradley, J. V. (1958). Complete counterbalancing of
immediate sequential effects in a latin square de-
sign. Journal of the American Statistical Association,
53(282):525–528.
Braun, D. and Matthes, F. (2019). Towards a framework for
classifying chatbots. In ICEIS (1), pages 496–501.
De Bra, P. M. and Post, R. (1994). Information retrieval
in the world-wide web: Making client-based search-
ing feasible. Computer Networks and ISDN Systems,
27(2):183–192.
Hart, S. and Staveland, L. (1988). Development of nasa-
tlx (task load index): Results and theoretical research,
human mental workload.
Hinderks, A., Schrepp, M., and Thomaschewski, J. (2018).
A benchmark for the short version of the user experi-
ence questionnaire. In WEBIST, pages 373–377.
Kaushik, A. (2019). Dialogue-based information retrieval.
In European Conference on Information Retrieval,
pages 364–368. Springer.
Kaushik, A., Bhat Ramachandra, V., and Jones, G. J. F.
(2020). An interface for agent supported conver-
sational search. In Proceedings of the 2020 Con-
ference on Human Information Interaction and Re-
trieval, CHIIR ’20, page 452–456, New York, NY,
USA. Association for Computing Machinery.
Kaushik, A. and Jones, G. J. (2021). A conceptual frame-
work for implicit evaluation of conversational search
interfaces. Mixed-Initiative ConveRsatiOnal Systems
workshop at ECIR 2021, pages 363–374.
Kaushik, A. and Jones, G. J. F. (2018). Exploring cur-
rent user web search behaviours in analysis tasks to be
supported in conversational search. In Second Inter-
national Workshop on Conversational Approaches to
Information Retrieval (CAIR’18), July 12, 2018, Ann
Arbor Michigan, USA.
Krathwohl, D. R. (2002). A revision of bloom’s taxonomy:
An overview. Theory into practice, 41(4):212–218.
Krogsæter, M., Oppermann, R., and Thomas, C. G. (1994).
A user interface integrating adaptability and adap-
tivity. Adaptive User Support. Ergonomic Design
of Manually and Automatically Adaptable Software,
pages 97–125.
Landis, J. R. and Koch, G. G. (1977). The measurement of
observer agreement for categorical data. biometrics,
pages 159–174.
Laugwitz, B., Held, T., and Schrepp, M. (2008). Construc-
tion and evaluation of a user experience questionnaire.
In Symposium of the Austrian HCI and usability engi-
neering group, pages 63–76. Springer.
Lewis, J. R. (1995). Ibm computer usability satisfac-
tion questionnaires: psychometric evaluation and in-
structions for use. International Journal of Human-
Computer Interaction, 7(1):57–78.
Maes, P. (1994). Agents that reduce work and information
overload. Communications of the ACM, 37(7):30–40.
McTear, M., Callejas, Z., and Griol, D. (2016). Conversa-
tional interfaces: Past and present. In The Conversa-
tional Interface, pages 51–72. Springer.
Nagarhalli, T. P., Vaze, V., and Rana, N. (2020). A review of
current trends in the development of chatbot systems.
In 2020 6th International Conference on Advanced
Computing and Communication Systems (ICACCS),
pages 706–710. IEEE.
Nogueira, R. and Cho, K. (2017). Task-oriented query
reformulation with reinforcement learning. arXiv
preprint arXiv:1704.04572.
Radlinski, F. and Craswell, N. (2017). A theoretical frame-
work for conversational search. In Proceedings of the
2017 Conference on Conference Human Information
Interaction and Retrieval, pages 117–126. ACM.
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M.,
Liu, Y., Xu, J., Ott, M., Shuster, K., Smith, E. M.,
et al. (2020). Recipes for building an open-domain
chatbot. arXiv preprint arXiv:2004.13637.
Schrepp, M. (2018). User experience questionnaire.
Schrepp, M., Hinderks, A., and Thomaschewski, J. (2017).
Design and evaluation of a short version of the user
experience questionnaire (ueq-s). IJIMAI, 4(6):103–
108.
Singh, A., Ramasubramanian, K., and Shivam, S. (2019).
Introduction to microsoft bot, rasa, and google di-
alogflow. In Building an Enterprise Chatbot, pages
281–302. Springer.
Stein, A. and Thiel, U. (1993). A conversational model of
multimodal interaction. GMD.
Trippas, J. R., Spina, D., Cavedon, L., Joho, H., and Sander-
son, M. (2018). Informing the design of spoken con-
versational search: Perspective paper. In Proceedings
of the 2018 Conference on Human Information Inter-
action & Retrieval, CHIIR ’18, pages 32–41, New
York, NY, USA. ACM.
Comparing Conventional and Conversational Search Interaction Using Implicit Evaluation Methods
303