Reading Students’ Multiple Mental States in Conversation from Facial and Heart Rate Cues

Shimeng Peng, Shigeki Ohira, Katashi Nagao

2020

Abstract

Students’ mental states have been widely acknowledged as crucial components for inferring their learning processes and are closely linked with learning outcomes. Understanding students’ complex mental states including concentration, confusion, frustration, and boredom in teacher-student conversation could benefit a human teacher’s perceptual and real-time decision-making capability in providing personalized and adaptive support in coaching activities. Many lines of research have explored the automatic measurement of students’ mental states in pre-designed human-computer tasks. It still remains a challenge to detect the complex mental states of students in real teacher-student conversation. In this study, we made such an attempt by describing a system for predicting the complex mental states of students from multiple perspectives: facial and physiological (heart rate) cues in real student-teacher conversation scenarios. We developed an advanced multi-sensor-based system and applied it in small-scale meetings to collect students’ multimodal conversation data. We demonstrate a multimodal analysis framework. Machine learning models were built by using extracted interpretable proxy features at a fine-grained level to validate their predictive ability regarding students’ multiple mental states. Our results provide evidence of the potential value of fusing multimodal data to understand students’ multiple mental states in real-world student-teacher conversation.

Paper Citation

in Harvard Style

Peng S., Ohira S. and Nagao K. (2020). Reading Students’ Multiple Mental States in Conversation from Facial and Heart Rate Cues.In Proceedings of the 12th International Conference on Computer Supported Education - Volume 1: CSEDU, ISBN 978-989-758-417-6, pages 68-76. DOI: 10.5220/0009564000680076

in Bibtex Style

@conference{csedu20,
author={Shimeng Peng and Shigeki Ohira and Katashi Nagao},
title={Reading Students’ Multiple Mental States in Conversation from Facial and Heart Rate Cues},
booktitle={Proceedings of the 12th International Conference on Computer Supported Education - Volume 1: CSEDU,},
year={2020},
pages={68-76},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0009564000680076},
isbn={978-989-758-417-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Computer Supported Education - Volume 1: CSEDU,
TI - Reading Students’ Multiple Mental States in Conversation from Facial and Heart Rate Cues
SN - 978-989-758-417-6
AU - Peng S.
AU - Ohira S.
AU - Nagao K.
PY - 2020
SP - 68
EP - 76
DO - 10.5220/0009564000680076