Restoring Eye Contact to the Virtual Classroom with Machine Learning

Ross Greer, Shlomo Dubnov

2021

Abstract

Nonverbal communication, in particular eye contact, is a critical element of the music classroom, shown to keep students on task, coordinate musical flow, and communicate improvisational ideas. Unfortunately, this nonverbal aspect to performance and pedagogy is lost in the virtual classroom. In this paper, we propose a machine learning system which uses single instance, single camera image frames as input to estimate the gaze target of a user seated in front of their computer, augmenting the user’s video feed with a display of the estimated gaze target and thereby restoring nonverbal communication of directed gaze. The proposed estimation system consists of modular machine learning blocks, leading to a target-oriented (rather than coordinate-oriented) gaze prediction. We instantiate one such example of the complete system to run a pilot study in a virtual music classroom over Zoom software. Inference time and accuracy meet benchmarks for videoconferencing applications, and quantitative and qualitative results of pilot experiments include improved success of cue interpretation and student-reported formation of collaborative, communicative relationships between conductor and musician.

Paper Citation

in Harvard Style

Greer R. and Dubnov S. (2021). Restoring Eye Contact to the Virtual Classroom with Machine Learning. In Proceedings of the 13th International Conference on Computer Supported Education - Volume 1: CSME, ISBN 978-989-758-502-9, pages 698-708. DOI: 10.5220/0010539806980708

in Bibtex Style

@conference{csme21,
author={Ross Greer and Shlomo Dubnov},
title={Restoring Eye Contact to the Virtual Classroom with Machine Learning},
booktitle={Proceedings of the 13th International Conference on Computer Supported Education - Volume 1: CSME,},
year={2021},
pages={698-708},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010539806980708},
isbn={978-989-758-502-9},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 13th International Conference on Computer Supported Education - Volume 1: CSME,
TI - Restoring Eye Contact to the Virtual Classroom with Machine Learning
SN - 978-989-758-502-9
AU - Greer R.
AU - Dubnov S.
PY - 2021
SP - 698
EP - 708
DO - 10.5220/0010539806980708