PHYSICAL TASK LEARNING SUPPORT SYSTEM
VISUALIZING A VIRTUAL TEACHER
BY MIXED REALITY
Tomoo Inoue and Motofumi Nakanishi
Graduate School of Library, Information and Media Studies, University of Tsukuba, Kasuga 1-2, Tsukuba, 305-8550, Japan
Keywords: Physical task learning, Mixed reality, Virtual reality, Virtual teacher, Interactive, View angle.
Abstract: To support learning physical task, we have developed a system showing a CG virtual teacher in the real
world by mixed reality technology. Most of existing physical task learning support systems do not show
instructions in the real world. They do not instruct interactively in real time, either. The proposed system
achieves these. Initial experiment of the effect of the location of the virtual teacher was also conducted.
1 INTRODUCTION
Research on supporting various tasks that utilizes
VR or MR (Mixed Reality) technology has been
actively conducted recently. Because it is known
very effective to use actual equipment in the real
environment for learning physical task, researches
on supporting physical task learning in such
environment using sensors and VR have been
conducted (Watanuki, 2007)(Ohsaki, 2005).
There are also researches on utilizing VR
technology for task learning or work support. For
example, there have been systems for learning body
movement (Yang, 2002)(Nakamura, 2003)(Honjou,
2005). These systems show a 3D CG teacher for a
learner to follow its movement. Other examples are
systems that show related information according to
the places in the real world. There have been work
support systems of this kind (Reiners,
1998)(Klinker, 2001).
MR is able to support tasks in the real world, and
is able to realize user interfaces that are more
adaptive to user behavior compared to VR (Bannai,
2003).
From these considerations, using MR is thought
to be suitable for supporting physical task learning.
Thus we have developed a task learning support
system using MR It visualizes a virtual 3D teacher in
the real world in front of a learner by MR, and is
interactive to the learner’s movement. In this paper,
we also discuss places of presenting the virtual
teacher.
Related
research is described in Section 2.
Proposal of a MR physical task learning support
system is given in Section 3. Implementation of the
system is described in Section 4. Preliminary
experiment is given in Section 5. Conclusion can be
found in Section 6.
2 RELATED RESEARCH
2.1 VR-based Task Learning Support
There have been various researches on VR-based
skill / task learning support. In the case of the VR-
based operation training system by Ohsaki et al., a
virtual work environment is built in the virtual space
(Ohsaki, 2005). VR-based dance training system by
Nakamura et al. shows 3D dance examples in the
virtual world (Nakamura, 2003). Sport skill
acquisition support system by Honjo et al. shows
examples in HMD (Honjou, 2005).
These VR-based learning support systems use
the virtual world. The virtual world is useful because
it can provide the environment that is difficult to
realize in the real world. On the contrary, it is
needed to reproduce the real world in the virtual
world to represent physical objects. It is not easy to
achieve (Bannai, 2003). MR technology has been
applied to deal with this problem.
In the study of the presentation position for a
body motion in a VR-based motion learning system,
both a model motion and a participant motion were
displayed in the same HMD screen (Kimura, 2007).
In this condition, it is easier to follow to the motion
276
Inoue T. and Nakanishi M. (2010).
PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY.
In Proceedings of the 2nd International Conference on Computer Supported Education, pages 276-281
DOI: 10.5220/0002794702760281
Copyright
c
SciTePress
when more axes between the model body and the
participant body are aligned. Because the participant
cannot see the real world in the condition, he/she
cannot see his/her own body. It may not be realistic
to learn body motion in this situation.
2.2 MR-based Task Support
The work support for instrument operation and the
work support for design in industrial field have been
researched using the MR. AR Power Plant
Maintenance overlays task instruction to the place of
maintenance through a HMD (Klinker, 2001). Space
Design helps designing 3D objects that are projected
on a desktop with a stylus pen (Fiorentino, 2002). As
a collaborative work support between distant places,
sharing an object and sharing the manipulations to
the object between the places has been researched
(Bannai, 2007).
In the MR-based task support, virtual objects are
overlaid in the physical world. So task support using
the physical equipment is possible. Gestural user
interfaces are also easy to use (Bannai, 2003).
2.3 MR-based Task Learning Support
Different from the task support, information
feedback to a learner is needed in the task learning
support (Watanuki, 2007). There are few MR-based
task learning support systems. Visualization of a
task record in MR space is an example of a MR-
based task learning support system. This proposes
efficient task review by visualizing the manipulation
record, position record, and the video of a target
object together (Miyasa, 2006). However this is to
support review of a task, not to support the task itself,
and does not give information feedback in real time.
3 PROPOSAL
3.1 System Proposal
In this research, the motion of a task is presented by
a 3D teacher model in front of a learner in the real
world.
There are various elements for describing tasks.
Some of them are the form, the position, the order,
the rhythm, the spending time, and so on. This
research deals with the form, the position, and the
order. Pushing buttons on a desk in the
predetermined order was chosen as the model task
because this is simple and generic physical task.
3.2 Context-aware Presentation
Appropriate information feedback is important for
effective and smooth task learning (Watanuki,
2007). In the conventional task learning by a real
teacher, he/she observes a learner and makes
intervention when the learner makes a mistake. To
realize such interactive information feedback to the
learner, sensing learning tasks or their progresses
can be seen in VR based learning support systems
(Watanuki, 2007)(Ohsaki, 2005)(Nakamura, 2003).
This is applicable to the MR-based learning support
systems. In our research, the learning tasks and their
progresses are sensed as learner’s motion and
operation to the target objects by the motion tracking
system. The teacher model is presented according to
them. This realizes interactivity in the real world.
3.3 Teacher Model Presentation
The position and direction of the teacher model is
known to be effective to the task understanding in
VR-based systems (Fiorentino, 2002). Because the
teacher model is presented over the real world in the
MR condition, its position and direction are not less
important than the VR condition. However, it has
not been known yet. To investigate appropriate
presentation of a teacher model, relative position
between a learner and the teacher model was
organized first in this section.
Let d be the distance between the model and the
learner. Let θ1 be the angle between the front
direction of the learner and the model. Let θ2 be the
rotation angle of the model from the front (Figure 1).
Figure 1: Teacher model presentation.
Y
X
Learne
r
θ
1
θ
2
d
PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY
277
Figure 2: Examples of relative positions of the teacher model.
Then relative position between a learner and the
teacher model can be represented by d, θ1, and θ2.
When presenting the teacher model, presenting the
teacher model behind the learner is not realistic.
Human horizontal view angle is about 200° at a
maximum. From these, - 100° < θ1 < 100° is natural
constraints (Kiyokawa, 2007). Here let height of
presentation be the same as the height of the eyes of
a learner. Examples of relative positions of the
teacher model are shown in Figure 2. They include
opposite direction, same direction, and same
position.
4 SYSTEM
4.1 Configuration
The system configuration is shown in Figure 3.
Three optical motion tracking cameras are placed
above a desk. These cameras detect markers that are
worn in the hand. Three markers that are at vertices
of a small triangle are used as a hand tracking
markers to know the hand direction stably. The
teacher model is presented by a video see-through
head mounted display. The size of the HMD is 640 x
480 pixels. The view angle of the HMD is H51°x
V37°. A position sensor for the HMD is Polhemus
FASTRAK. There are 5 buttons on the desktop.
These buttons are simple push buttons and do not
contain any sensor or mechanism. Pushing the
buttons is generic example of physical task.
4.2 Use Scenario
The system architecture is shown in Figure 4. Scene
of using the system and the learner’s view are shown
in Figure 5 and Figure 6, respectively.
Before the system starts, a learner wears the
HMD on his/her head and the markers on his/her
hand. When the system starts, it displays the teacher
model overlaid to the physical space. The teacher
model shows the physical motion of a task. In the
physical space typically are physical objects to be
handled.
In the current example case of the system,
Figure 3: System configuration.
Figure 4: System architecture.
they are 5 push buttons on a desk of approximately
12 cm in diameter. The learner can learn the
physical task or the physical motion that the teacher
model presents by watching it in front of the
physical target objects.
Motion tracking camera
HMD
Task status recognition
Learner’s behavio
r
Task
Teache
r
model presentation
User Interface
Button
HMD
Marker
Motion tracking camera
Y
X
d0
θ
2
X
Y
θ
2
d
θ
1
θ
2
180°
Y
X
θ
1
θ
2
d
Y
X
θ
1
θ
2
d
CSEDU 2010 - 2nd International Conference on Computer Supported Education
278
Figure 5: Using the system.
Figure 6: Learner’s view.
The learner can learn by following the motion as
shown in Figure 6. The motion to be learnt is given
to the system in advance so that the system is able to
recognize whether the learner’s motion is correct or
not. When the learner’s motion is correct, the system
continues to present the next motion. When the
learner’s motion is not correct, the system presents
the same motion again. In the example, the position
of the leaner’s hand is tracked by the motion
tracking cameras. It is checked that the learner
pushed the same button the teacher model pushed in
the example.
4.3 Appearance and Motion of the
Teacher Model
To avoid effects of the teacher model appearance on
the task performance, a plain cylindrical 3D model is
used (Figure 7). The model is animated by the free
software RokDeBone2 (RokDeBone2, 2010). The
motion
of the current system consists of a sequence
of
the motion units. Each motion unit which shows
Figure 7: Appearance of the teacher model.
pushing each button was prepared in advance. Then
the motion unit is combined in a sequence. Different
task is produced by the different combination of the
motion units.
5 PRELIMINARY EXPERIMENT
OF THE TEACHER MODEL
POSITION
5.1 Objective
This system can change the position of the teacher
model, but which position is desirable to a learner is
not known.
Because the position of the teacher model can be
set freely and its variation is unlimited, we need to
get a clue to the possible positions before elaborate
experiment. So the objective of this preliminary
experiment is to narrow down candidate positions.
5.2 Procedure
The teacher model was presented at various
positions. Intuitive understandability of the motion
was rated in 7-point scale by 2 participants. Total of
120 positions were evaluated where 3 conditions of
d (0m, 1m, 2m), 7 conditions of θ1 (0°, ±30°, ±45°,
±60°), and 8 conditions of θ2 (0°, ±45°, ± 90 °, ±
135 °, 180°). Figure 8 shows these positions where
the participant is located at the origin.
5.3 Result
The positions that were rated 4 or more out of 7 are
shown in Figure 9. As for the distance d, many of
PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY
279
Figure 8: Evaluated positions in the preliminary
experiment.
Figure 9: Highly evaluated positions.
them are near the participant. As for θ1, many of
them are in front of the participant, but not exact
front. As for θ2, side view of the teacher model
seems to be preferred.
6 CONCLUSIONS
A physical task learning system that uses interactive
MR teacher is proposed. It provides instruction in
the real world which is different from VR-based
systems. It also provides real time interactive
support of a physical task by sensing the learner’s
motion. Visualization of a teacher model will be
researched more in the future.
ACKNOWLEDGEMENTS
This research was partially supported by the
Research Project Grant of Graduate School of
Library, Information and Media Studies, University
of Tsukuba.
REFERENCES
Watanuki, K., 2007. Knowledge Acquisition and Job
Training for Fundamental Manufacturing
Technologies and Skills by Using Immersive Virtual
Environment. Journal of Japanese Society for
Artificial Intelligence, Vol.22, No.4, pp.480-490.
Ohsaki, J., Matsubara, Y., Iwane, N., and Nakamura, M.,
2005. VR-based Learning Support System for
Operator Training : Design and Evaluation of Basic
System, IEICE technical report. Education technology,
Vol.105, No.337, pp.1-6.
Yang, U., Kim, C. J., 2002. Implementation and
Evaluation of "Just Follow Me": an Immersive, VR-
based, Motion Training System, Presence:
Teleoperators and Virtual Environments, Vol.11, No.3,
pp.304-323.
Nakamura, A., Niwayama, T., Murakami, T., Tabata, S.,
and Kuno, Y., 2003. Analysis of Motions and
Development of Application Systems for Traditional
Dances, IPSJ SIG Technical Reports, Vol.2003, No.36,
pp.85-92.
Honjou, N., Isaka, T., Mitsuda, T., and Kawamura, S.,
2005. Proposal of Method of Sports Skill Learning
using HMD, Transactions of Virtual Reality Society of
Japan, Vol.10, No.1, pp.63-70.
Reiners, D., Stricker, D., Klinker, G., and Muller, S., 1998.
Augmented Reality for Construction Tasks: Doorlock
Assembly, In. Proc. of IWAR’98, pp.31-46.
Klinker, G., Creighton, O., Dutoit, A. H., Kobylinski, R.,
Vilsmeier, C., and Brugge, B., 2001. Augmented
maintenance of powerplants: A prototyping case study
of a mobile ARsystem, In. Proc. of ISAR’01, pp.124-
132.
Bannai, Y. and Yamamoto, H., 2003. Mixed Reality
Systems for Supporting Spatial Work, IPSJ SIG
Technical Reports, Vol.2003, No.106, pp.19-24.
Kimura, A., Kuroda, T., Manabe, Y., and Chihara, K.,
2007. A Study of Display of Visualization of Motion
Instruction Supporting, Japan Journal of Educational
Technology, Vol.30, No.4, pp.293-303.
Fiorentino, M., De Amicis, R., Monno, G., and Stork, A.,
2002. A Mixed Reality Workspace for Aesthetic
Industrial Design, In. Proc. ISMAR’02, pp.86-94.
Bannai, Y., Tamaki, H., Suzuki, Y., Shigeno, H. and
Okada, K., 2007. Remote Collaborative Mixed Reality
Using Tangible Objects, IPSJ Journal, Vol.48, No.7,
pp.2645-2476.
Miyasa, K., Bannai, H., Shigeno, H., and Okada, K., 2006.
A Recording and Visualization Method of Working
CSEDU 2010 - 2nd International Conference on Computer Supported Education
280
Activities in a Mixed Reality Space, IPSJ Journal,
Vol.47, No.1, pp.181-192.
Kiyokawa, K., 2007. Visual Display Technology in
Virtual Reality, Journal of Japan Society for Fuzzy
Theory and Intelligent Informatics, Vol.19, No.4,
pp.318-325.
RokDeBone2, http://www5d.biglobe.ne.jp/~ochikko/ro
kdebone.htm
PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY
281