Feedback Design in Multimodal Dialogue Systems

Peter Van Rosmalen, Dirk Börner, Jan Schneider, Olga Petukhova, Joy van Helvert


This paper discusses the design and development of the instructional aspects of a multimodal dialogue system to train youth parliament members’ presentation and debating skills. Real-time, in-action feedback informs learners on the fly how they perform key skills and enables them to adapt instantly. About-action feedback informs learners after finishing a task how they perform key skills and enables them to monitor their progress and adapt accordingly in subsequent tasks. In- and about-action feedback together support the enhancement of the learners’ metacognitive skills, such as self-monitoring, self-regulation and self-reflection thus reflect in- and about action. We discusses the theoretical considerations of the feedback, the type of data available and different ways to analyse and combine them, the timing of feedback and, finally, provide an instructional design blueprint giving a global outline of a set of tasks with stepwise increasing complexity and the feedback proposed. We conclude with the results of the first experiment with the system focussing on non-verbal communication skills.


  1. Argyle, M., 1994. The psychology of interpersonal behaviour. Penguin Books, London.
  2. Bjerregaard, M. and Compton, E., 2011. Public Speaking Handbook. Snow College, Suppliment for Public Speaking.
  3. Brooke, J., 1996. SUS: a "quick and dirty" usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, and A. L. McClelland. Usability Evaluation in Industry. London: Taylor and Francis.
  4. Coninx, N., Kreijns, K. and Jochems, W., 2013. The use of keywords for delivering immediate performance feedback on teacher competence development. European Journal of Teacher Education, 36(2), 164- 182.
  5. DeVito J.A., 2014. The Essential Elements of Public Speaking. Pearson.
  6. D'Souza, C., 2013. Debating: a catalyst to enhance learning skills and competencies. Education + Training, Vol. 55 Iss 6 pp. 538 - 549.
  7. Duncan, S., 1970. Towards a grammar for floor apportionment: A system approach to face-to-face interaction. In Proceedings of the Second Annual Environmental Design Research Association Conference, pages 225-236, Philadelphia.
  8. Engeser, S. and Rheinberg, F., 2008. Flow, performance and moderators of challenge-skill balance. Motiv Emot 32: 158-172.
  9. Hattie, J., and Timperley, H., 2007. The power of feedback. Review of Educational Research, 77(1), 81- 112.
  10. Heylen, D., 2006. Head gestures, gaze and the principles of conversational structure. International journal of Humanoid Robotics, 3(3):241-267.
  11. Kendon, A., 2004. Gesture: visible action as utterance. Cambridge University Press, Cambridge.
  12. Kiili, K., De Freitas, S., Arnab S. and Lainemac, T., 2012 The Design Principles for Flow Experience in Educational Games. Procedia Computer Science 15 (2012) 78 - 91. Virtual Worlds for Serious Applications (VS-GAMES'12).
  13. Lewis, J. R., and Sauro, J., 2009 The Factor Structure of the System Usability Scale. Proceedings of the first Human Centered Design Conference HCD 2009, San Diego, 2009.
  14. Mann W. and Thompson, S., 1988. Rhetorical Structure Theory 2/rst.pdf.
  15. Nicol D. and Macfarlane-Dick D., 2006. "Formative assessment and self-regulated learning: A model and seven principles of good feedback practice" Studies in Higher Education vol.31 no.2 pp.199-218.
  16. Nielsen, J. and Landauer, T., K., 1993 A mathematical model of the finding of usability problems. Proceedings of ACM INTERCHI'93 Conference, pp. 206-213.
  17. Nye, B.D., Graesser, A.C. and Hu. X., 2014. AutoTutor and Family: A Review of 17 Years of Natural Language Tutoring. International Journal Artificial Intelligence Education, 24, pp 427-469.
  18. Peldszus, A. and Stede M. 2013. From argument diagrams to argumentation mining in texts: a survey. International Journal of Cognitive Informatics and Natural Intelligence (IJCINI) 7(1), 1-31.
  19. Petukhova, V., 2005. Multidimensional interaction of multimodal dialogue acts in meetings. MA thesis,. Tilburg University.
  20. Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A. and Webber, B., (2008). The Penn Discourse Treebank 2.0. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC). Marrakech, Morocco.
  21. Scheflen, A., 1964. The significance of posture in communication systems. Psychiatry, 17:316-331.
  22. Schneider, J., Börner, D., Van Rosmalen, P. and Specht, M., 2014. Presentation Trainer: A Toolkit for Learning Non-verbal Public Speaking Skills. In Proceedings of the 9th European Conference on Technology Enhanced Learning, EC-TEL 2014. Open Learning and Teaching in Educational Communities, Lecture Notes in Computer Science Volume 8719, 2014, pp 522-525. Springer International Publishing.
  23. Schön, D. A., 1983. The Reflective Practitioner: How Professionals Think in Action. (T. Smith, Ed.) (p. 374). Basic Books.
  24. Sweller, J., 1994. "Cognitive Load Theory, learning difficulty, and instructional design". Learning and Instruction 4 (4): 295-312.
  25. Triantafyllou, E., Timcenko, O., and Triantafyllidis, G., 2014. Reflections on Students' Projects with Motion Sensor Technologies in a Problem-Based Learning Environment. Proceedings of the 8th European Conference on Games Based Learning ECGBL 2014. Berlin, 2014.
  26. Trimboli, A., and Walker, M. B., 1987. Nonverbal dominance in the communication of affect: A myth? Journal of Nonverbal Behavior, 11(3) pp. 180-190.
  27. Van Merriënboer, J. J. G. and Kirschner, P. A., 2013. Ten Steps to Complex Learning (2nd Rev. Ed.). New York: Routledge.
  28. Van Rosmalen, P., Boyle, E.A., Van der Baaren, J., Kärki, A.I. and Del Blanco Aguado, A., 2014. A case study on the design and development of mini-games for research methods and statistics. EAI Endorsed Transactions on Game Based Learning 14(3): e5.

Paper Citation

in Harvard Style

Van Rosmalen P., Börner D., Schneider J., Petukhova O. and van Helvert J. (2015). Feedback Design in Multimodal Dialogue Systems . In Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-108-3, pages 209-217. DOI: 10.5220/0005423102090217

in Bibtex Style

author={Peter Van Rosmalen and Dirk Börner and Jan Schneider and Olga Petukhova and Joy van Helvert},
title={Feedback Design in Multimodal Dialogue Systems},
booktitle={Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU,},

in EndNote Style

JO - Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Feedback Design in Multimodal Dialogue Systems
SN - 978-989-758-108-3
AU - Van Rosmalen P.
AU - Börner D.
AU - Schneider J.
AU - Petukhova O.
AU - van Helvert J.
PY - 2015
SP - 209
EP - 217
DO - 10.5220/0005423102090217