First Experiments on Interaction Quality Modelling for Human-Human Conversation

Anastasiia Spirina, Maxim Sidorov, Roman Sergienko, Alexander Schmitt

2016

Abstract

This work presents the first experimental results on Interaction Quality modelling for human-human conversation, as an adaptation of the Interaction Quality metric for human-computer spoken interaction. The prediction of an Interaction Quality score can be formulated as a classification problem. In this paper we describe the results of applying several classification algorithms such as: Kernel Naïve Bayes Classifier, k-Nearest Neighbours algorithm, Logistic Regression, and Support Vector Machines, to a data set. Moreover, we compare the results of modelling for two approaches for Interaction Quality labelling and consider the results depending on different emotion sets. The results of Interaction Quality modelling for human-human conversation may be used both for improving the service quality in call centres and for improving Spoken Dialogue Systems in terms of flexibility, user-friendliness and human-likeness.

References

  1. Bailey, R. A., 2008. Design of comparative experiments, Cambridge University Press.
  2. le Cessie, S., van Houwelingen, J. C., 1992. Ridge Estimators in Logistic Regression. Applied Statistics, Vol. 41, No. 1, pp. 191-201.
  3. Cristianini, N., Shawe-Taylor, J., 2000. An introduction to Support Vector Machines and Other Kernel-based Learning Methods, Cambridge University Press.
  4. Eyben, F., Weninger, F., Gross, F., Schuller, B., 2013. Recent Developments in openSMILE, the Munich Open-Source Multimedia Feature Extractor. In Proceedings of ACM Multimedia (MM), pp. 835-838.
  5. Goutte, C., Gaussier, E., 2005. A probabilistic interpretation of precision, recall and f-score, with implication for evaluation. Advances in information retrieval. Springer.
  6. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I. H., 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations, Vol. 11, No. 1.
  7. John, G.H., Langley, P., 1995. Estimating Continuous Distributions in Bayesian Classifiers. In Eleventh Conference on Uncertainty in Artificial Intelligence, San Mateo, pp. 338-345.
  8. Kennedy, J. J., Bush, A. J., 1985. An introduction to the design and analysis of experiments in behavioral research, University Press of America.
  9. Llimona, Q., Luque, J., Anguera, X., Hidalgo, Z., Park, S., Oliver, N., 2015. Effect of gender and call duration on customer satisfaction in call center big data. In Proceedings of Interspeech, pp. 18525-1829.
  10. Pallotta, V., Delmonte, R., Vrieling, L., Walker, D., 2011. Interaction Mining: the new frontier of Call Center Analytics. In CEUR Workshop Proceedings.
  11. Park, Y., Gates, S. C., 2009. Towards real-time measurement of customer satisfaction using automatically generated call transcripts. In Proceedings of the 18th ACM conference on Information and knowledge management, New York, pp. 1387-1396.
  12. Platt, J., 1998. Sequential Minimal Optimisation: A Fast Algorithm for Training Support Vector Machines. TechReport MSR-TR-98-14, Microsoft Research.
  13. Rafaeli, A., Ziklik, L., Doucet, L., 2008. The Impact of Call Center Employees' Customer Orientation Behaviors on Service Quality. Journal of Service Research, Vol. 10, No. 3, pp. 239-255.
  14. R Core Team, 2015. R: A language and environment for statistical computing, R Foundation for Statistical Computing. Vienna, Austria. http://www.rproject.org/.
  15. Schmitt, A., Schatz, B., Minker, W., 2011. Modeling and predicting quality in spoken human-computer interaction. In Proceedings of the SIGDIAL 2011 Conference. Association for Computational Linguistics, pp. 173-184.
  16. Schmitt, A., Ultes, S., Minker, W., 2012. A Parameterized and Annotated Corpus of the CMU Let's Go Bus Information System. In International Conference on Language Resources and Evaluation (LREC).
  17. Schmitt, A., Ultes, S., 2015. Interaction Quality: Assessing the quality of ongoing spoken dialog interaction by experts - And how it relates to user satisfaction. Speech Communication, Vol. 74, pp. 12 - 36.
  18. Spirina, A. V., Sidorov M. Yu., Sergienko, R. B., Semenkin E. S., Minker, W., 2016. Human-Human Task-Oriented Conversations Corpus for Interaction Quality Modelling. Vestnik SibSAU, Vol. 17, No. 1.
  19. Ultes, S., ElChabb, R., Minker, W., 2012. Application and Evaluation of a Conditioned Hidden Markov Model for Estimating Interaction Quality of Spoken Dialogue Systems. In Proceedings of the 4th International Workshop On Spoken Dialogue Systems (IWSDS), pp. 141-150.
  20. Witten, I. H., Frank, E., Hall, M. A., 2011. Data mining: practical machine learning tools and techniques, Morgan Kaufmann. USA, 3rd edition.
Download


Paper Citation


in Harvard Style

Spirina A., Sidorov M., Sergienko R. and Schmitt A. (2016). First Experiments on Interaction Quality Modelling for Human-Human Conversation . In Proceedings of the 13th International Conference on Informatics in Control, Automation and Robotics - Volume 2: ICINCO, ISBN 978-989-758-198-4, pages 374-380. DOI: 10.5220/0005983103740380


in Bibtex Style

@conference{icinco16,
author={Anastasiia Spirina and Maxim Sidorov and Roman Sergienko and Alexander Schmitt},
title={First Experiments on Interaction Quality Modelling for Human-Human Conversation},
booktitle={Proceedings of the 13th International Conference on Informatics in Control, Automation and Robotics - Volume 2: ICINCO,},
year={2016},
pages={374-380},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005983103740380},
isbn={978-989-758-198-4},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 13th International Conference on Informatics in Control, Automation and Robotics - Volume 2: ICINCO,
TI - First Experiments on Interaction Quality Modelling for Human-Human Conversation
SN - 978-989-758-198-4
AU - Spirina A.
AU - Sidorov M.
AU - Sergienko R.
AU - Schmitt A.
PY - 2016
SP - 374
EP - 380
DO - 10.5220/0005983103740380