PHYSICAL TASK LEARNING SUPPORT SYSTEM

VISUALIZING A VIRTUAL TEACHER

BY MIXED REALITY

Tomoo Inoue and Motofumi Nakanishi

Graduate School of Library, Information and Media Studies, University of Tsukuba, Kasuga 1-2, Tsukuba, 305-8550, Japan

Keywords: Physical task learning, Mixed reality, Virtual reality, Virtual teacher, Interactive, View angle.

Abstract: To support learning physical task, we have developed a system showing a CG virtual teacher in the real

world by mixed reality technology. Most of existing physical task learning support systems do not show

instructions in the real world. They do not instruct interactively in real time, either. The proposed system

achieves these. Initial experiment of the effect of the location of the virtual teacher was also conducted.

1 INTRODUCTION

Research on supporting various tasks that utilizes

VR or MR (Mixed Reality) technology has been

actively conducted recently. Because it is known

very effective to use actual equipment in the real

environment for learning physical task, researches

on supporting physical task learning in such

environment using sensors and VR have been

conducted (Watanuki, 2007)(Ohsaki, 2005).

There are also researches on utilizing VR

technology for task learning or work support. For

example, there have been systems for learning body

movement (Yang, 2002)(Nakamura, 2003)(Honjou,

2005). These systems show a 3D CG teacher for a

learner to follow its movement. Other examples are

systems that show related information according to

the places in the real world. There have been work

support systems of this kind (Reiners,

1998)(Klinker, 2001).

MR is able to support tasks in the real world, and

is able to realize user interfaces that are more

adaptive to user behavior compared to VR (Bannai,

2003).

From these considerations, using MR is thought

to be suitable for supporting physical task learning.

Thus we have developed a task learning support

system using MR It visualizes a virtual 3D teacher in

the real world in front of a learner by MR, and is

interactive to the learner’s movement. In this paper,

we also discuss places of presenting the virtual

teacher.

research is described in Section 2.

Proposal of a MR physical task learning support

system is given in Section 3. Implementation of the

system is described in Section 4. Preliminary

experiment is given in Section 5. Conclusion can be

found in Section 6.

2 RELATED RESEARCH

2.1 VR-based Task Learning Support

There have been various researches on VR-based

skill / task learning support. In the case of the VR-

based operation training system by Ohsaki et al., a

virtual work environment is built in the virtual space

(Ohsaki, 2005). VR-based dance training system by

Nakamura et al. shows 3D dance examples in the

virtual world (Nakamura, 2003). Sport skill

acquisition support system by Honjo et al. shows

examples in HMD (Honjou, 2005).

These VR-based learning support systems use

the virtual world. The virtual world is useful because

it can provide the environment that is difficult to

realize in the real world. On the contrary, it is

needed to reproduce the real world in the virtual

world to represent physical objects. It is not easy to

achieve (Bannai, 2003). MR technology has been

applied to deal with this problem.

In the study of the presentation position for a

body motion in a VR-based motion learning system,

both a model motion and a participant motion were

displayed in the same HMD screen (Kimura, 2007).

In this condition, it is easier to follow to the motion

276

Inoue T. and Nakanishi M. (2010).

PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY.

In Proceedings of the 2nd International Conference on Computer Supported Education, pages 276-281

DOI: 10.5220/0002794702760281

 SciTePress

when more axes between the model body and the

participant body are aligned. Because the participant

cannot see the real world in the condition, he/she

cannot see his/her own body. It may not be realistic

to learn body motion in this situation.

2.2 MR-based Task Support

The work support for instrument operation and the

work support for design in industrial field have been

researched using the MR. AR Power Plant

Maintenance overlays task instruction to the place of

maintenance through a HMD (Klinker, 2001). Space

Design helps designing 3D objects that are projected

on a desktop with a stylus pen (Fiorentino, 2002). As

a collaborative work support between distant places,

sharing an object and sharing the manipulations to

the object between the places has been researched

(Bannai, 2007).

In the MR-based task support, virtual objects are

overlaid in the physical world. So task support using

the physical equipment is possible. Gestural user

interfaces are also easy to use (Bannai, 2003).

2.3 MR-based Task Learning Support

Different from the task support, information

feedback to a learner is needed in the task learning

support (Watanuki, 2007). There are few MR-based

task learning support systems. Visualization of a

task record in MR space is an example of a MR-

based task learning support system. This proposes

efficient task review by visualizing the manipulation

record, position record, and the video of a target

object together (Miyasa, 2006). However this is to

support review of a task, not to support the task itself,

and does not give information feedback in real time.

3 PROPOSAL

3.1 System Proposal

In this research, the motion of a task is presented by

a 3D teacher model in front of a learner in the real

world.

There are various elements for describing tasks.

Some of them are the form, the position, the order,

the rhythm, the spending time, and so on. This

research deals with the form, the position, and the

order. Pushing buttons on a desk in the

predetermined order was chosen as the model task

because this is simple and generic physical task.

3.2 Context-aware Presentation

Appropriate information feedback is important for

effective and smooth task learning (Watanuki,

2007). In the conventional task learning by a real

teacher, he/she observes a learner and makes

intervention when the learner makes a mistake. To

realize such interactive information feedback to the

learner, sensing learning tasks or their progresses

can be seen in VR based learning support systems

(Watanuki, 2007)(Ohsaki, 2005)(Nakamura, 2003).

This is applicable to the MR-based learning support

systems. In our research, the learning tasks and their

progresses are sensed as learner’s motion and

operation to the target objects by the motion tracking

system. The teacher model is presented according to

them. This realizes interactivity in the real world.

3.3 Teacher Model Presentation

The position and direction of the teacher model is

known to be effective to the task understanding in

VR-based systems (Fiorentino, 2002). Because the

teacher model is presented over the real world in the

MR condition, its position and direction are not less

important than the VR condition. However, it has

not been known yet. To investigate appropriate

presentation of a teacher model, relative position

between a learner and the teacher model was

organized first in this section.

Let d be the distance between the model and the

learner. Let θ1 be the angle between the front

direction of the learner and the model. Let θ2 be the

rotation angle of the model from the front (Figure 1).

Figure 1: Teacher model presentation.

Learne

PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY

277

Figure 2: Examples of relative positions of the teacher model.

Then relative position between a learner and the

teacher model can be represented by d, θ1, and θ2.

When presenting the teacher model, presenting the

teacher model behind the learner is not realistic.

Human horizontal view angle is about 200° at a

maximum. From these, - 100° < θ1 < 100° is natural

constraints (Kiyokawa, 2007). Here let height of

presentation be the same as the height of the eyes of

a learner. Examples of relative positions of the

teacher model are shown in Figure 2. They include

opposite direction, same direction, and same

position.

4 SYSTEM

4.1 Configuration

The system configuration is shown in Figure 3.

Three optical motion tracking cameras are placed

above a desk. These cameras detect markers that are

worn in the hand. Three markers that are at vertices

of a small triangle are used as a hand tracking

markers to know the hand direction stably. The

teacher model is presented by a video see-through

head mounted display. The size of the HMD is 640 x

480 pixels. The view angle of the HMD is H51°x

V37°. A position sensor for the HMD is Polhemus

FASTRAK. There are 5 buttons on the desktop.

These buttons are simple push buttons and do not

contain any sensor or mechanism. Pushing the

buttons is generic example of physical task.

4.2 Use Scenario

The system architecture is shown in Figure 4. Scene

of using the system and the learner’s view are shown

in Figure 5 and Figure 6, respectively.

Before the system starts, a learner wears the

HMD on his/her head and the markers on his/her

hand. When the system starts, it displays the teacher

model overlaid to the physical space. The teacher

model shows the physical motion of a task. In the

physical space typically are physical objects to be

handled.

In the current example case of the system,

Figure 3: System configuration.

Figure 4: System architecture.

they are 5 push buttons on a desk of approximately

12 cm in diameter. The learner can learn the

physical task or the physical motion that the teacher

model presents by watching it in front of the

physical target objects.

Motion tracking camera

HMD

Task status recognition

Learner’s behavio

Task

Teache

model presentation

User Interface

Button

HMD

Marker

Motion tracking camera

d＝0

＝0°

＝180°

＝0°

CSEDU 2010 - 2nd International Conference on Computer Supported Education

278

Figure 5: Using the system.

Figure 6: Learner’s view.

The learner can learn by following the motion as

shown in Figure 6. The motion to be learnt is given

to the system in advance so that the system is able to

recognize whether the learner’s motion is correct or

not. When the learner’s motion is correct, the system

continues to present the next motion. When the

learner’s motion is not correct, the system presents

the same motion again. In the example, the position

of the leaner’s hand is tracked by the motion

tracking cameras. It is checked that the learner

pushed the same button the teacher model pushed in

the example.

4.3 Appearance and Motion of the

Teacher Model

To avoid effects of the teacher model appearance on

the task performance, a plain cylindrical 3D model is

used (Figure 7). The model is animated by the free

software RokDeBone2 (RokDeBone2, 2010). The

motion

of the current system consists of a sequence

the motion units. Each motion unit which shows

Figure 7: Appearance of the teacher model.

pushing each button was prepared in advance. Then

the motion unit is combined in a sequence. Different

task is produced by the different combination of the

motion units.

5 PRELIMINARY EXPERIMENT

OF THE TEACHER MODEL

POSITION

5.1 Objective

This system can change the position of the teacher

model, but which position is desirable to a learner is

not known.

Because the position of the teacher model can be

set freely and its variation is unlimited, we need to

get a clue to the possible positions before elaborate

experiment. So the objective of this preliminary

experiment is to narrow down candidate positions.

5.2 Procedure

The teacher model was presented at various

positions. Intuitive understandability of the motion

was rated in 7-point scale by 2 participants. Total of

120 positions were evaluated where 3 conditions of

d (0m, 1m, 2m), 7 conditions of θ1 (0°, ±30°, ±45°,

±60°), and 8 conditions of θ2 (0°, ±45°, ± 90 °, ±

135 °, 180°). Figure 8 shows these positions where

the participant is located at the origin.

5.3 Result

The positions that were rated 4 or more out of 7 are

shown in Figure 9. As for the distance d, many of

PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY

279

Figure 8: Evaluated positions in the preliminary

experiment.

Figure 9: Highly evaluated positions.

them are near the participant. As for θ1, many of

them are in front of the participant, but not exact

front. As for θ2, side view of the teacher model

seems to be preferred.

6 CONCLUSIONS

A physical task learning system that uses interactive

MR teacher is proposed. It provides instruction in

the real world which is different from VR-based

systems. It also provides real time interactive

support of a physical task by sensing the learner’s

motion. Visualization of a teacher model will be

researched more in the future.

ACKNOWLEDGEMENTS

This research was partially supported by the

Research Project Grant of Graduate School of

Library, Information and Media Studies, University

of Tsukuba.

REFERENCES

Watanuki, K., 2007. Knowledge Acquisition and Job

Training for Fundamental Manufacturing

Technologies and Skills by Using Immersive Virtual

Environment. Journal of Japanese Society for

Artificial Intelligence, Vol.22, No.4, pp.480-490.

Ohsaki, J., Matsubara, Y., Iwane, N., and Nakamura, M.,

2005. VR-based Learning Support System for

Operator Training : Design and Evaluation of Basic

System, IEICE technical report. Education technology,

Vol.105, No.337, pp.1-6.

Yang, U., Kim, C. J., 2002. Implementation and

Evaluation of "Just Follow Me": an Immersive, VR-

based, Motion Training System, Presence:

Teleoperators and Virtual Environments, Vol.11, No.3,

pp.304-323.

Nakamura, A., Niwayama, T., Murakami, T., Tabata, S.,

and Kuno, Y., 2003. Analysis of Motions and

Development of Application Systems for Traditional

Dances, IPSJ SIG Technical Reports, Vol.2003, No.36,

pp.85-92.

Honjou, N., Isaka, T., Mitsuda, T., and Kawamura, S.,

2005. Proposal of Method of Sports Skill Learning

using HMD, Transactions of Virtual Reality Society of

Japan, Vol.10, No.1, pp.63-70.

Reiners, D., Stricker, D., Klinker, G., and Muller, S., 1998.

Augmented Reality for Construction Tasks: Doorlock

Assembly, In. Proc. of IWAR’98, pp.31-46.

Klinker, G., Creighton, O., Dutoit, A. H., Kobylinski, R.,

Vilsmeier, C., and Brugge, B., 2001. Augmented

maintenance of powerplants: A prototyping case study

of a mobile ARsystem, In. Proc. of ISAR’01, pp.124-

132.

Bannai, Y. and Yamamoto, H., 2003. Mixed Reality

Systems for Supporting Spatial Work, IPSJ SIG

Technical Reports, Vol.2003, No.106, pp.19-24.

Kimura, A., Kuroda, T., Manabe, Y., and Chihara, K.,

2007. A Study of Display of Visualization of Motion

Instruction Supporting, Japan Journal of Educational

Technology, Vol.30, No.4, pp.293-303.

Fiorentino, M., De Amicis, R., Monno, G., and Stork, A.,

2002. A Mixed Reality Workspace for Aesthetic

Industrial Design, In. Proc. ISMAR’02, pp.86-94.

Bannai, Y., Tamaki, H., Suzuki, Y., Shigeno, H. and

Okada, K., 2007. Remote Collaborative Mixed Reality

Using Tangible Objects, IPSJ Journal, Vol.48, No.7,

pp.2645-2476.

Miyasa, K., Bannai, H., Shigeno, H., and Okada, K., 2006.

A Recording and Visualization Method of Working

CSEDU 2010 - 2nd International Conference on Computer Supported Education

280

Activities in a Mixed Reality Space, IPSJ Journal,

Vol.47, No.1, pp.181-192.

Kiyokawa, K., 2007. Visual Display Technology in

Virtual Reality, Journal of Japan Society for Fuzzy

Theory and Intelligent Informatics, Vol.19, No.4,

pp.318-325.

RokDeBone2, http://www5d.biglobe.ne.jp/~ochikko/ro

kdebone.htm

PHYSICAL TASK LEARNING SUPPORT SYSTEM VISUALIZING A VIRTUAL TEACHER BY MIXED REALITY

281