BELONG: Body Experienced Learning Support System

based on Gesture Recognition

Enhancing the Sense of Immersion in a Dinosaurian Environment

Mikihiro Tokuoka

, Haruya Tamaki

, Tsugunosuke Sakai

, Hiroshi Mizoguchi

, Ryohei Egusa

2,3

Shigenori Inagaki

, Mirei Kawabata

, Fusako Kusunoki

and Masanori Sugimoto

Department of Mechanical Engineering, Tokyo University of Science, 2641 Yamazaki, Noda-shi, Chiba-ken, Japan

JSPS Research Fellow, Tokyo, Japan

Graduate School of Human Development and Environment, Kobe University, Hyogo, Japan

Department of Computing, Tama Art University, Tokyo, Japan

Hokkaido University, Hokkaido, Japan

Keywords: Kinect V2 Sensor, Immersive, Learning Support System, Body Movements.

Abstract: As the first step toward realizing an immersive learning support system for museums, Yoshida et al. developed

and evaluated a prototype system. However, this system was problematic in that it could only be operated by

using simple body movements. Moreover, the other problem was that learning about paleontology itself

cannot be performed only by learning about a paleontological environment. Therefore, we developed an

immersive learning support system "BELONG" as an upgraded version of the above-mentioned system. Using

a recognizer capable of gesture recognition, the system can be operated using complicated body movements.

The improved system enables learners to enhance their sense of immersion in a paleontological environment

and learn about the fossil itself and its paleontology. This paper summarizes the prototype of "BELONG" and

describes the experiments that were performed to evaluate its ability to achieve learning support and

immersion.

1 INTRODUCTION

Museums are important places for children to learn

about science (Falk, J. H., 2012). They also operate

as centers for informal education in connection with

schools, and they enhance the effectiveness of

scientific education (Stocklmayer, S. M., 2010).

However, because the main learning method within

museums is the study of the specimens on display and

their explanations, there are few opportunities for

learners to observe or experience the environment

about which they are learning. In particular, it is

impossible to experience a paleontological

environment, which includes extinct animals and

plants, and their ecological environment (Adachi, T.,

2013). It is difficult for children to learn about such

environments merely by looking at fossils and

listening to commentary. Overcoming this problem

would qualitatively improve scientific learning

within museums. As for these problems, a system that

simulates a paleontological environment and

transitions that would be impossible to experience in

reality would solve the problem. The system would

also need to enhance learners’ sense of immersion.

Therefore, such a system is needed.

In order to enhance the sense of immersion, a full

body interaction interface in which the movement of

the whole body is linked to the operation of the

system was previously shown to be effective

(Klemmer S., 2006). Yoshida et al. also developed a

system targeting a full body interaction interface;

however, the system could only be operated using

simple body movements and this was problematic

(Yoshida, R., 2015). Moreover, learning about

paleontology itself cannot be performed only by

learning of a paleontological environment.

Therefore, we developed an immersive learning

support system "BELONG" as an upgraded version

of the original system by Yoshida et al. The upgraded

system solves the above problems. Compared with

the conventional learning support system, using a

recognizer capable of gesture recognition, the system

is operated using complicated body movements.

Tokuoka, M., Tamaki, H., Sakai, T., Mizoguchi, H., Egusa, R., Inagaki, S., Kawabata, M., Kusunoki, F. and Sugimoto, M.

BELONG: Body Experienced Learning Support System based on Gesture Recognition - Enhancing the Sense of Immersion in a Dinosaurian Environment.

DOI: 10.5220/0006357104870492

In Proceedings of the 9th International Conference on Computer Supported Education (CSEDU 2017) - Volume 1, pages 487-492

ISBN: 978-989-758-239-4

487

Moreover, body movements are incorporated as part

of the observation behavior of a fossil exhibition,

which previously consisted of a conventional written

explanation. The system enables learners to enhance

their sense of immersion in a paleontological

environment and learn about the fossil itself and its

paleontology.

In this paper, we summarize the prototype of

"BELONG" as the first step toward developing the

immersive learning support system for the fossil

exhibition at the museum. In addition, we describe the

results of our experimental evaluation of the learning

support and immersion abilities of the system with the

aim of clarifying whether it can provide learners with

a realistic paleontological observation experience.

2 LEARNING SUPPORT SYSTEM

2.1 Belong

We aim to realize the immersive learning support

system "BELONG" that simulates a paleontological

environment and transitions that are impossible to

experience in reality for efficient learning at the

museum. Figure 1 illustrates the concept of

"BELONG." This system accepts body movements as

input for observational behavior. The movements of

the whole body and the system operation are linked;

therefore, it is possible to enhance the sense of

immersion in the paleontological environment. The

sense of immersion improves if the system can be

operated in conjunction with complicated body

movements as compared with a case in which the

system is operated with simple body movements. The

recognition of complicated body movements should

not involve attaching expensive sensors or devices to

learners when implementing it in a museum. In this

system, we utilize Microsoft’s Kinect v2 sensor, a

range-image sensor originally developed as a home

videogame device. Because BELONG comprises

only a Kinect v2 sensor, projector, and control PC, it

allows us to provide a low-cost immersive learning

experience within a small space. The advantage of

this arrangement is that it is possible to easily change

the learning contents. Moreover, we recognize the

body movements of learners by gesture recognition

using the Kinect v2 sensor. The gesture recognition

system, which can also interpret complicated body

movements, registers the body movement the creator

wishes to recognize and judges whether it is

recognized by verifying the similarity with the body

movement.

Figure 1: Concept of BELONG.

2.2 Configuration of the System

We developed an immersive learning support system

"BELONG" that simulates a paleontological

environment and transitions that are impossible to

experience in reality.

As a first step towards the

realization of this system, we are developing a system

to simulate paleoecology, especially learning about

dinosaurs, based on experiences that simulate a

paleontological excavation.

Our assumption was that

learners' interest would increase by virtually

excavating fossils included in the current exhibition.

However, because excavation motions are complex

body movements, gesture recognition was used.

(Tokuoka, M., 2017) When the excavation proceeds

successfully, videos showing the characteristics of the

dinosaur are displayed.

Linking the body and the

video in this way increases the sense of immersion.

These body movements are recognized by a Kinect v2

sensor, the properties of which are described below.

Microsoft’s Kinect v2 sensor is a range-image

sensor originally developed as a home videogame

device. Although it is inexpensive, the sensor can

record sophisticated measurements regarding the

user’s location. Additionally, this sensor can

recognize humans and the human skeleton using the

library in Kinect’s software development kit for

Windows. Kinect can measure the location of human

body parts such as hands and legs, and it can identify

the user’s pose or status with this function and the

location information. Moreover, Kinect Studio and

Visual Gesture Builder are used to recognize

complicated body movements captured by the Kinect

sensor. These enable complicated body movements to

be recognized using the discriminator (Tokuoka, M.,

2017). By using these, it is possible to create a

discriminator that registers the body movements we

want to recognize and can accurately recognize body

movements using machine learning. As a complicated

CSEDU 2017 - 9th International Conference on Computer Supported Education

488

body movement, this time we adopted the movements

associated with performing an excavation.

Figure 2 shows that this system consists of the

above Kinect V2 sensor, a control PC, and a projector.

As shown in Figure 3, the current system enables

children to learn about five dinosaurs, namely

Tyrannosaurs, Tambaryu, Archeoptery, Pteranodon,

and Ichthiosaurus. This system starts operation when

learners stand in front of the screen. First they select

the dinosaur fossil of which they want to learn by

using a pushing motion of the palm toward the screen.

After selection, learners excavate the corresponding

dinosaur fossil. We recognize the excavation

movement in this step using the Kinect v2 sensor.

When the excavation succeeds with the excavation

movement at full power, the video showing the

characteristics of dinosaurs is shown. This video

shows the habitat of the dinosaur in the Paleozoic era

and the size of the animal. Our evaluation aimed to

recognize whether complicated body movements

such as excavation are recognized by the

discriminator of the subsystem. (Tokuoka, M., 2017)

We determined that movements associated with

excavation are recognized when extensive and

accurate movements are made at full power. If

learners are shy and make a small movement, it is not

recognized as an excavation movement and fails.

Learners concentrate further by moving their bodies

towards the screen with enthusiasm, increasing the

sense of immersion in the paleontological

environment. In the museum, fossils are mere

sightings; however, they felt that the virtual

excavation enhanced their participation by

visualizing the virtual world.

After the excavation movement succeeds, you can

conduct separate learning for the five dinosaurs. This

is followed by an explanation of five contents. Figure

4 (a), (b), (c), (d), and (e) show the operation for each

of the contents.

Figure 2: Setup of the system.

Figure 3: Details of dinosaurs.

(a)Tyrannosaurs

Learners can answer quizzes about the tyrannosaurs

in this content by using a pushing motion of the palm

toward the screen. There are three quizzes. A

commentary video follows when learners answer

correctly. An incorrect answer allows learners to

make another selection. By learning about dinosaurs

in the form of quizzes, they can learn while having

fun.

(b)Tambaryu

Tambaryu moves sideways following learners’

movement to either the right or the left. Thus,

Tambaryu moves in response to their movement,

thereby increasing their sense of immersion in the

paleontological environment.

(c)Archeoptery

Learners can feed an Archeoptery by using the hand

action of gripping and opening. First, they can select

the bait of the Archeoptery displayed on the screen by

using a hand gripping action. They then move this

bait toward the mouth of the Archeoptery and open

their hands. Then, the Archeoptery eats the bait.

Therefore, they virtually experience feeding and they

know what the Archeoptery ate.

(d)Pteranodon

When learners wave their hands in a greatly animated

way a Pteranodon approaches.

A video of the

Pteranodon approaching is shown, and they seem to

be attacked by it. However, in the video the

Pteranodon does not have muscular strength as it

takes learners away. This enables them to learn about

the physical features of dinosaurs.

(e)Ichthiosaurus

When learners move forward and backward, they can

learn about the physical features of the Ichthiosaurus.

When they approach the screen, a video showing the

BELONG: Body Experienced Learning Support System based on Gesture Recognition - Enhancing the Sense of Immersion in a

Dinosaurian Environment

489

state of the stomach is shown. When they retreat from

the screen, the video shows the full appearance of the

body. Thus, their sense of distance changes in

conjunction with moving forward and backwards.

This enables them to learn about the size and shape of

the Ichthiosaurus.

As an observation behavior of paleontology, this

system can learn while experiencing contents

accompanying five kinds of different physical

behaviors using the above-mentioned technique of

gesture recognition.

By performing more complicated body

movements than would be possible with conventional

systems, we were able to develop a full-body

interactive interface that is more immersive. The use

of different actions for each dinosaur enables learners

to learn while having fun. Moreover, learners can

learn about five dinosaurs themselves and clearly

understand the difference between these dinosaurs.

(a)

(b)

(c)

(d)

(e)

Figure 4: Five contents.

3 EVALUATION

3.1 Methodology

Participants: Twenty-two fifth grade students (9 boys

and 13 girls) from elementary schools attached to the

national university corporation.

Location: H Prefectural Museum of Natural History

Topic: The participants each experienced the system

individually, after which they evaluated the system.

The evaluation method was used to obtain their

CSEDU 2017 - 9th International Conference on Computer Supported Education

490

feedback using a seven-point Likert scale

questionnaire on paper; the values on the Likert scale

correspond to “strongly agree,” “agree,” “somewhat

agree,” “neither agree nor disagree,” “somewhat

disagree,” “disagree,” and “strongly disagree,” from

seven to one, respectively. The questionnaire

structure was based on the

physical/emotional/narrative presence scale (PENS).

The questionnaire consisted of eight statements in

response to which the participants rated their

agreement: “When I was moving my body and

playing the game to learn about dinosaurs, I felt like I

was in a time or place different from my usual

environment,” “The details of the ancient time made

me feel like I was actually having an adventure in that

time,” “What happened within the system (creatures

and their movements) did not inspire me (reverse

score question),” “The system experience was

fascinating,” “I was able to experience the ancient

time just as deeply as the real world,” “During my

system experience, I felt as though I became part of

the time when dinosaurs lived,” “I felt very satisfied

with the system experience,” and “I felt that the

ancient scenery, creatures, and the way they lived on

the screen were very real.”

Test Date: 11/26/2016

3.2 Results

Responses for “strongly agree,” “agree,” and

“somewhat agree” were grouped as positive

responses; those for “neither agree nor disagree,”

“somewhat disagree,” “disagree,” and “strongly

disagree” were grouped as neutral/negative

responses. The number of positive and

neutral/negative responses was analyzed using

Fisher’s exact test with a 1 × 2 contingency table. The

responses for the reverse score question were

reversed.

Table 1 summarizes the system evaluations. The

positive responses exceeded neutral/negative

responses for all eight questions. Additionally, there

were significant differences at the 1% level between

the number of positive and neutral/negative responses

for seven of the statements: “When I was moving my

body and playing the game to learn about dinosaurs,

I felt like I was in a time or place different from my

usual environment,” “The details of the ancient time

made me feel like I was actually having an adventure

in that time,” “What happened within the system

(creatures and their movements) did not inspire me

(reverse score question),” “The system experience

was fascinating,” “I was able to experience the

ancient time just as deeply as the real world,” “I felt

very satisfied with the system experience,” and “I felt

that the ancient scenery, creatures, and the way they

lived on the screen were very real.”

Furthermore, there was a significant difference at

the 5% level between the number of positive and

Table 1: Subjective evaluation of the DINOU experience: physical/emotional/narrative presence.

7 6 5 4 3 2 1

When I was moving my body and playing the game to learn

about dinosaurs, I felt like I was in a time or place different

from my usual environment

4 10 4 1 2 0 1

The details of the ancient time made me feel like I was actually

having an adventure in that time

2 8 7 3 0 1 1

What happened within the system (creatures and their

movements) did not inspire me (-)

1 1 1 5 2 7 5

4. The system experience was fascinating

11 4 5 1 0 0 1

I was able to experience the ancient time just as deeply as the

real world

1 6 9 3 2 0 1

During my system experience, I felt as though I became a part

of the time when dinosaurs lived

2 5 8 2 3 0 2

7. I felt very satisfied with the system experience

7 9 3 2 0 0 1

I felt that the ancient scenery, creatures, and the way they lived

on the screen were very real

4 6 6 3 2 0 1

N = 22

p <.01,

p <.05 (-): Reverse score question

7 = Strongly agree; 6 = Agree; 5 = Somewhat agree; 4 = Neither agree nor disagree; 3 = Somewhat disagree;

2 = Disagree; 1 = Strongly disagree

BELONG: Body Experienced Learning Support System based on Gesture Recognition - Enhancing the Sense of Immersion in a

Dinosaurian Environment

491

neutral/negative responses for the statement: “During

my system experience, I felt as though I became a part

of the time when dinosaurs lived.”

4 CONCLUSIONS

In this paper, we proposed a prototype of "BELONG"

as the first step of the immersive learning support

system at the fossil exhibition at the museum. This

system comprises a full body interaction interface

using a Kinect v2 sensor as a gesture discriminator.

We used an experimental evaluation to assess the

extent to which the system supported learning and

immersion.

The results of the evaluation experiments revealed

that learners experienced reality, presence, and appeal

for paleoecology through experiencing this system

using complicated body movements. The results

showed that virtual observation behavior of

paleontology using the gesture recognition ability of

"BELONG" enabled learners to experience reality.

This indicates that "BELONG" is effective as a

method capable of providing a place of observation to

learn about aspects of paleontology such as dinosaurs

where a direct observational experience would be

impossible.

ACKNOWLEDGEMENTS

This work was supported in part by Grants-in-Aid for

Scientific Research (B). The evaluation was

supported by the Museum of Nature and Human

Activities, Hyogo, Japan.

REFERENCES

Falk, J. H., Dierking, L. D., 2012. Museum Experience

Revisited. Left Coast Press. Walnut Creek California,

USA, 2

edition.

Stocklmayer, S. M., Rennie, L. J., Gilbert, J. K. 2010. The

roles of the provision of effective science education,

Studies in Science Education, 46(1), pp. 1-44,

Routledge Journals, Taylor & Francis Ltd.

Adachi, T., Goseki, M., Muratsu, K., Mizoguchi, H.,

Namatame, M., Sugimoto, M., Takeda, Y., Fusako, K.,

Estuji, Y., Shigenori, I., 2013. Human SUGOROKU:

Full-body interaction system for students to learn

vegetation succession. Proceedings of the 12th

International Conference on Interaction Design and

Children. pp. 364-367, ACM.

Klemmer S., Hartmann B., Takayama L., How bodies

matter: five themes for interaction design, DIS '06

Proceedings of the 6th conference on Designing

Interactive systems, pp. 140-149.

Yoshida, R., Egusa, R., Saito, M., Namatame, M.,

Sugimoto, M., Kusunoki, F., Etsuji, Y., Shigenori, I.,

Yoshiaki, T., Mizoguchi, H., 2015. BESIDE: Body

Experience and Sense of Immersion in Digital

Paleontological Environment. Proceedings of the 33rd

Annual ACM Conference Extended Abstracts on

Human Factors in Computing Systems. pp. 1283-1288.

ACM.

Tokuoka, M., Tamaki, H., Sakai, T., Mizoguchi, H., Egusa,

R., Inagaki, S., Kawabata, M., Kusunoki, F., Sugimoto,

M., 2017. Development of Gesture Recognition Sub-

system for BELONG. Proceedings of the 9th

International Conference on Computer Supported

Education, CSEDU 2017. To appeared.

CSEDU 2017 - 9th International Conference on Computer Supported Education

492