EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources

Juan Manuel Lucas-Cuesta, Fernando Fernández-Martínez, G. Dragos Rada, Syaheerah L. Lutfi, Javier Ferreiros

Abstract

We present an evaluation of a spoken language dialogue system with a module for the management of user-related information, stored as user preferences and privileges. The flexibility of our dialogue management approach, based on Bayesian Networks (BN), together with a contextual information module, which performs different strategies for handling such information, allows us to include user information as a new level into the Context Manager hierarchy. We propose a set of objective and subjective metrics to measure the relevance of the different contextual information sources. The analysis of our evaluation scenarios shows that the relevance of the short-term information (i.e. the system status) remains pretty stable throughout the dialogue, whereas the dialogue history and the user profile (i.e. the middle-term and the long-term information, respectively) play a complementary role, evolving their usefulness as the dialogue evolves.

References

  1. Callejas, Z. and López-Cózar, R. (2008). Relations between de-facto criteria in the evaluation of a spoken dialogue system. Speech Communication, 50:646-665.
  2. Chin, D. (2001). Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and UserAdapted Interaction, 11:181-194.
  3. Dybkjaer, L., Bernsen, N., and Minker, W. (2004). Evaluation and usability of multimodal spoken language dialogue systems. Speech Communication, 43:33-54.
  4. Fernández, F., Blázquez, J., Ferreiros, J., Barra, R., MacíasGuarasa, J., and Lucas-Cuesta, J. (2008). Evaluation of a Spoken Dialogue System for Controlling a Hifi Audio System. In Proceedings of the IEEE Workshop on Spoken Language Technology (SLT08), pages 137- 140.
  5. Fernández, F., Ferreiros, J., Sama, V., Montero, J., SanSegundo, R., and Macías-Guarasa, J. (2005). Speech Interface for Controlling a Hi-Fi Audio System Based on a Bayesian Belief Networks Approach for Dialog Modeling. In Proceedings of the 9th European Conference on Speech Communication and Technology (INTERSPEECH05), pages 3421-3424.
  6. Gena, C. (2005). Methods and Techniques for the Evaluation of User-Adaptive Systems. The Knowledge Engineering Review, 20(1):1-37.
  7. Litman, D. and Pan, S. (2002). Designing and Evaluating an Adaptive Spoken Dialogue System. User Modeling and User-Adapted Interaction, 12:111-137.
  8. Lucas-Cuesta, J., Fernández, F., Salazar, J., Ferreiros, J., and San-Segundo, R. (2009). Managing Speaker Identity and User Profiles in a Spoken Dialogue System. Sociedad Espan˜ola de Procesamiento de Lenguaje Natural (SEPLN), 43:77-84. ISSN: 1135-5948.
  9. Möller, S., Smeele, P., Boland, H., and Krebber, J. (2007). Evaluating Spoken Dialogue Systems According to De-Facto Standards: a Case Study. Computer, Speech and Language, 21:26-53.
  10. Walker, M., Litman, D., Kamm, C., and Abella, A. (1997). PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In Proceedings ACL/EACL, pages 271-280.
  11. Zukerman, I. and Litman, D. (2001). Natural Language Processing and User Modeling: Synergies and Limitations. User Modeling and User-Adapted Interaction, 11:129-158.
Download


Paper Citation


in Harvard Style

Manuel Lucas-Cuesta J., Fernández-Martínez F., Dragos Rada G., L. Lutfi S. and Ferreiros J. (2011). EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources . In Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART, ISBN 978-989-8425-40-9, pages 218-223. DOI: 10.5220/0003140302180223


in Bibtex Style

@conference{icaart11,
author={Juan Manuel Lucas-Cuesta and Fernando Fernández-Martínez and G. Dragos Rada and Syaheerah L. Lutfi and Javier Ferreiros},
title={EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources},
booktitle={Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,},
year={2011},
pages={218-223},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003140302180223},
isbn={978-989-8425-40-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Conference on Agents and Artificial Intelligence - Volume 1: ICAART,
TI - EVALUATION OF A USER-ADAPTED SPOKEN LANGUAGE DIALOGUE SYSTEM - Measuring the Relevance of the Contextual Information Sources
SN - 978-989-8425-40-9
AU - Manuel Lucas-Cuesta J.
AU - Fernández-Martínez F.
AU - Dragos Rada G.
AU - L. Lutfi S.
AU - Ferreiros J.
PY - 2011
SP - 218
EP - 223
DO - 10.5220/0003140302180223