ial. Indeed, (Sigrist et al., 2012) deeply investigated
the impact of feedback modalities for motor learning
in VE and Real Environment (RE). Those modalities
must be carefully chosen according to the complex-
ity of the task and the cognition abilities of the learn-
ers. Nonetheless a feedback cannot be reduced to its
modality, and other design elements are crucial such
as, its virtual representation, its triggering rule or the
motion metric to monitor. For this last point, even
if the teacher is involved in the metric initial choice,
the efficiency of the evaluation system is not guaran-
teed (Senecal et al., 2002). Consequently, a system
must be built to allow any teacher, without IT knowl-
edge, to (re)design and (re)implement all feedback
elements, to make them efficient and adapted to the
learning situation. To our knowledge, no such a sys-
tem exists, except the work of (Lo et al., 2019), lim-
ited to one VE not built for learning purposes.
This paper proposes a three-dimensional descrip-
tive model (virtual representation, triggering rules, in-
volved 3D objects) of a pedagogical feedback and
its operationalization through the GEstural FEedback
EDitor (GEFEED). This editor allows any teacher
to create and integrate feedbacks, characterizing the
performed technical gestures, in any VLE developed
with the unity engine. Five kinds of feedbacks are
available (i.e. visual color, visual text, audio from a
file, audio from a text and haptic vibration) and can be
associated with a set of four kind of triggering rules
(i.e. time, contact between 3D objects, spatial config-
uration or threshold of a motion metric to reach).
Section 2 reviews the past studies regarding VLE
for gesture learning, feedbacks, their design and their
potential re-usability. The next section presents the
4 dimensional descriptive model. The architecture
and Human Computer Interface (HCI) of GEFEED
are described in section 4. A first VLE dedicated to
the dilution in biology is considered in section 5, with
the creation of a feedback example. Section 6 is dedi-
cated to an experiment where the teachers must create
their feedbacks with this VLE. The usability and use-
fulness of GEFEED, its main functonalities are put to
the test. The results of the experiment are discussed
in section 7 while perspectives end this paper.
2 RELATED WORKS
For learning gesture-based tasks or motor skills, var-
ious VLE have been built integrating real-time feed-
backs to: guide learners in correcting their motions
(Luo et al., 2011; Cannavò et al., 2018; Liu et al.,
2020), following the protocol made of an action se-
quence (Mahdi et al., 2019; Mizuyama, 2010), per-
forming a self-evaluation (Cannavò et al., 2018), im-
proving engagement (Adolf et al., 2019) or enhanc-
ing the overall pedagogical experience (Mizuyama,
2010).
All those VLE are, by design, specific to their ped-
agogical and research objectives including the pro-
vided feedbacks. In this work a pedagogical feed-
back is considered as a pedagogical information, pre-
viously defined by a teacher, provided to the learner
through a virtual representation, during the task or af-
ter it, to: (i) assist learners in the evaluation of the task
(ii), its progression or (iii), guide them in its good ex-
ecution. By defining the motion features of the 3D
object to monitor (e.g. geometric, kinematic or dy-
namic features, collisions, etc.), triggering rules (e.g.,
threshold for features, time step, etc.) and a virtual
representation with which the pedagogical informa-
tion will be conveyed (e.g. an arrow the motion must
follow, a hand vibration to avoid reaching a danger-
ous area, etc.), a strategy for operationalising the ped-
agogical information is defined. However, given the
learning context and the pedagogical objective, one
can ask for the best design strategy of such a feed-
back.
(Sigrist et al., 2012) investigated the impact of
feedback modalities (i.e. visual, haptic, audio, multi-
modal) for motor learning in VE and RE. Visual feed-
backs are mainly used, intuitive and efficient. A first
type of visual feedback relies on a color change, for
example, of specific joints of the body to help in ad-
justing its position and orientation to learn tai-chi, i.e.
green when the learner motion is close to the expert
one, red otherwise (Liu et al., 2020). In addition, a
textual score can be added that points out the body
position (correct or not) to assesse the overall perfor-
mance of a basketball throw (Cannavò et al., 2018).
A last recurrent type of visual feedbacks is the replay
of teachers’ motions through a 3D avatar and the mo-
tion trajectory, displayed during or after the perfor-
mance, to guide learners and for self-evaluation (Can-
navò et al., 2018; Le Naour et al., 2019; Djadja et al.,
2020).
Audio feedback can support visual ones as they
are easily interpretable (Sigrist et al., 2012). A first
strategy is to add recorded voices to displayed texts
advising learners to, for example, handle a Chinese
frying pan (Mizuyama, 2010) (e.g. "don’t move your
left wrist", "push the contents forward with the la-
dle"). A brief sound can also be heard for the comple-
tion of a good action or an inappropriate one. In the
context of tenon structure training, (Chen et al., 2019)
provided a collision sound when the hand touches a
tenon part. A prompt, read by a prerecorded voice,
can then deliver the related knowledge. Those feed-
A 3D Descriptive Model for Designing Multimodal Feedbacks in any Virtual Environment for Gesture Learning
85