Science Education Enhancement within a Museum using

Computer-human Interaction Technology

Haruya Tamaki

, Tsugunosuke Sakai

, Ryuichi Yoshida

, Ryohei Egusa

2,3

, Shigenori Inagaki

Etsuji Yamaguchi

, Fusako Kusunoki

, Miki Namatame

, Masanori Sugimoto

and Hiroshi Mizoguchi

Department of Mechanical Engineering, Tokyo University of Science, 2641 Yamazaki, Noda-shi, Chiba-ken, Japan

JSPS Research Fellow, Tokyo, Japan

Graduate School of Human Development and Environment, Kobe University, Hyogo, Japan

Department of Computing, Tama Art University, Tokyo, Japan

Tsukuba University of Technology, Ibaraki, Japan

Hokkaido University, Hokkaido, Japan

Keywords: Kinect Sensor, Museum, Immersive Learning Support System.

Abstract: We are developing an immersive learning support system for paleontological environments in museums. The

system measures the physical movement of the learner using a Kinect sensor, and provides a sense of

immersion in the paleontological environment by adapting the surroundings according to these movements.

As the first step toward realizing an immersive learning support system for museums, Yoshida et al. developed

and evaluated a prototype system. However, their system cannot learn about certain paleontological features

such as the names of extinct animals and their characteristics. Therefore, we developed an improved version

of this system that allows for an enriched knowledge of paleontological environments, focusing in particular

on extinct animals and plants and the ecological environment. Here, we evaluate the system’s learning

assistant and immersive features insofar as they are directed toward children in primary school. This paper

summarizes the current system and describes the evaluation results.

1 INTRODUCTION

Museums are important places for children to learn

about science (Falk and Dierking, 2012). They also

operate as centers for informal education in

connection with schools, and they enhance the

effectiveness of scientific education (Stocklmayer et

al., 2010). However, because the main learning

method within museums is the study of the specimens

on display and their explanations, there are few

chances for learners to observe or experience the

environment about which they are learning. In

particular, it is impossible to experience a

paleontological environment, which includes extinct

animals and plants and their ecological environment

(Adachi et al., 2013). It is difficult for children to

learn about such environments merely with fossils

and commentary. Overcoming this problem would

qualitatively improve scientific learning within

museums.

One solution to this problem would be some sort

of booth and video content that reproduced the

paleontological environment artificially. However,

issues pertaining to space and cost mean that most

museums cannot accommodate such an exhibit.

Hence, we are developing an immersive learning

system that will enable learners to explore a virtual

paleontological environment at any museum. This

“Body Experience and Sense of Immersion in a

Digital paleontological Environment” (or BESIDE)

system acquires information regarding the learner’s

movement using a Kinect sensor, and operates

according to this information. The system uses

multiple screens spread across the learner’s entire

field of vision. By being projected into this virtual

space, the learner can adopt physical movements as

observational actions. We expect that this will

engender a sense of immersion in the virtual space.

Because BESIDE comprises only a commercial

image sensor, projector, and control PC, we can

provide a low-cost immersive learning experience

within a small space.

As the first step toward realizing an immersive

learning support system for museums, Yoshida et al.

Tamaki, H., Sakai, T., Yoshida, R., Egusa, R., Inagaki, S., Yamaguchi, E., Kusunoki, F., Namatame, M., Sugimoto, M. and Mizoguchi, H.

Science Education Enhancement within a Museum using Computer-human Interaction Technology.

In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016) - Volume 2, pages 181-185

ISBN: 978-989-758-179-3

181

developed and evaluated a prototype system (Yoshida

et al., 2015).

Their system allows the learner to experience the

paleontological environment. By being projected in

such a paleontological environment, the learner can

adopt physical movements as observational actions.

This engenders a sense of immersion in the

paleontological environment. However, their system

is limited insofar as users cannot learn about these

paleontological environments, including the extinct

animals and plants and their ecological environment.

Because of this limitation, we developed an improved

version of the system developed by Yoshida et al.

With our proposed system, learners can develop a

richer knowledge of paleontological environments,

including extinct animals and plants and the

ecological environment.

In this paper, we summarize our improved version

of the system developed by Yoshida et al., and we

describe an evaluation of its learning support and

immersion.

2 IMMERSIVE LEARNING

SUPPORT SYSTEM

2.1 BESIDE

We are developing “BESIDE” as an immersive

learning system that enables learners to explore a

virtual paleontological environment at any museum.

BESIDE consists of various sensors and digital

learning content. The sensors measure the learners’

location, pose, and actions, and the learning content

is then controlled according to these measurements.

Figure 1 illustrates the concept of BESIDE.

Figure 1: Concept of BESIDE.

Learners walk around a space containing a screen

that displays a virtual environment, and observable

objects such as animals move in synchronization with

them. In this way, learners feel immersed in the

paleontological environment. Synchronizing the

movement of the paleontological animals with that of

the learner makes it possible to consider the animals

“real” in some sense, rather than imaginary.

Furthermore, by introducing near-real activities,

such as “approaching the observable object” or

“diving into water,” the level of interest and learning

effects are more enhanced than they would be by

merely attending a typical exhibition or watching a

video.

2.2 Current System

We are developing a system that includes a figure of

the learner in a virtual environment displayed on the

screen, allowing the learner to experience the

paleontological environment and learn about extinct

animals and plants and the ecological environment.

Figure 2 shows the setup of the current system.

Figure 2: Setup of the current system.

In the system, the learner first stands in front of

the screen, and a camera image of the learner is

displayed with the background removed. Learners

can change the background by moving their hands

and interacting with the content. The system then

allows learners to select animals appropriate for

different geologic eras, and these are then displayed

on the screen. Furthermore, the animal’s

characteristics are displayed in the screen. Learners

use their hands to select the animal’s name and then

interact with the content.

The system must be able to acquire real-time data

regarding the learner’s location and movements. Thus,

we utilize Microsoft’s Kinect sensor, a range-image

sensor originally developed as a home videogame

device. Although it is inexpensive, the sensor can

record sophisticated measurements regarding the

user’s location (Shotton, J., 2011). Additionally, this

sensor can recognize humans and the human skeleton

using the library in Kinect’s software development kit

for Windows. Kinect can measure the location of

human body parts such as hands and legs, and it can

CSEDU 2016 - 8th International Conference on Computer Supported Education

182

identify the user’s pose or status with this function

and the location information. We use these functions

to recognize humans and detect the human skeleton.

The system reproduces a paleontological

environment by placing a suitable animal from one of

three geologic eras (viz., Paleozoic, Mesozoic, and

Cenozoic) into the display. The animals appropriate

to each era are listed in Table 1. We also prepared

three kinds of fossils that are typical of these

animals—i.e., fossils that can be found at the museum.

Furthermore, we prepared animals’ names and

characteristics. With this system, information

regarding the frames and depth are measured with the

Kinect sensor. The measurements are sent to the PC,

and various images are displayed on the screen. The

system has the following functions:

(a) Displays images based on sensor information;

(b) Operates using the learner’s body motion;

animations.

(d) Teaches animal names and characteristics

Table 1: Details of animals.

Paleozoic Mesozoic Cenozoic

Elrathiakingi

(Trilobite)

Perisphinctina

(Ammonoidea)

Mammoth

Orthoceras Triceratops Merycoidodon

Crinoidea Mosasaurus Shark

Function (a) displays the user on the screen by

recognizing the user’s outline. Accordingly, the user

is placed into the paleontological environment.

Function (b) allows the system to operate according

to the user’s hand movements. The learner can click

a button by pushing his/her hand toward the Kinect

sensor. The learner then chooses one of the three

geologic eras, and the screen is replaced with an

image appropriate to that era. Icons of the animals are

displayed at the bottom the screen. If the learner

selects an animal from that geologic era, the animal is

displayed on the screen. If the selected animal is from

a different era, a warning sounds and the learner is

asked to choose again. This enables the system to

teach the learner about which animals lived in each

era, in addition to providing a sense of immersion.

Function (c) enables the displayed animal to move

around the screen as a GIF animation. By means of

this, learners can observe extinct animals as though

they were real. Function (d) displays animals’ names

at the bottom the screen. If the learner selects the

correct name of the animal, its characteristics are

displayed on the screen. If the selected animal is from

the wrong era, a warning sounds and the learner is

asked to choose again. By means of this, the user

learns about paleontological environments, including

extinct animals and plants and their ecological

environment. Figure 3 shows a learner using the

current systems’ functions.

Function (a)

Function (b)

Function (c)

Function (d)

Figure 3: Functions of the current system.

Figure 4: Using the current system.

Science Education Enhancement within a Museum using Computer-human Interaction Technology

183

3 EVALUATION

3.1 Method

Participants: The participants comprised 27 students

(age: 10–11 years) from an elementary school in

Kobe, Japan (12 boys and 15 girls).

Process: The participants were organized into groups

of 3–4, and were instructed to observe selected fossils

on display at the museum. These fossils were from

organisms available in the prototype system.

Participants used the system, one by one. The

participants were able to select the geologic era

relevant to the fossils they studied—viz., Paleozoic,

Mesozoic, and Cenozoic—and the paleoecology of

the era was then simulated. We prepared three

silhouette quizzes and three quizzes regarding animal

names for each geologic era. Each of the participants

spent approximately five minutes interacting with the

system. Finally, the participants evaluated the system.

Evaluation: A questionnaire was used to evaluate two

features of the system: learning assistance and

immersion. The questionnaire consisted of 12 items.

The learning assistance provided by the system was

evaluated with five questions regarding changes in

the understanding and interests of the participants

after experiencing the paleoecological simulation.

Immersion was evaluated with seven questions

(based on Player Experience of Need Satisfaction

Physical/Emotional/Narrative Presence Scale)

regarding the effectiveness of the system at enhancing

the immersive experience of the paleoecological

simulation and the learner’s projection into the

scenery. For each question, participants answered,

“strongly agree,” “agree,” “agree somewhat,”

“neutral,” “disagree,” “disagree somewhat,” or

“strongly disagree.”

Date of Experiment: July 25, 2015

Location: Natural Science Museum, Hyogo

Prefecture.

3.2 Results

The average scores for each quiz were as follows: the

silhouette quizzes resulted in an average of 2.68, and

the quizzes regarding animal names resulted in an

average of 2.82.

Data obtained from the questionnaire was divided

into two groups—positive evaluations (“strongly

agree,” “agree,” and “agree somewhat”), and neutral or

negative evaluations (“neutral,” “disagree,” “disagree

somewhat,” and “strongly disagree”)—and the results

for the two groups were totaled. The difference

between the positive and neutral/negative evaluations

was computed using Fisher’s Exact Test (1 x 2).

Table 2: Results of evaluation about learning assistance.

Item 7 6 5 4 3 2 1

When I accomplished all the quizzes in the system, I experienced genuine pride.

＊＊

17 5 6 0 0 0 0

When I read the explanations of the ancient organisms provided by the system, I believed

that they were effective at improving the learning experience at the museum.

＊＊

15 6 5 1 0 1 0

Using body motion in order to complete the quizzes was fun.

＊＊

20 5 3 0 0 0 0

The silhouette quizzes were effective at improving the learning experience at the museum.

＊＊

19 6 2 1 0 0 0

The quizzes regarding animal names were effective at improving the learning experience at

the museum.

＊＊

21 4 2 1 0 0 0

N=28

7:Strongly Agree, 6:Agree, 5:Agree Somewhat, 4:No Option, 3:Disagree Somewhat, 2:Disagree, 1:Strongly Disagree

＊＊

P<0.01

Table 3: Results from the Evaluation of the Proposed System’s Immersion.

Item 7 6 5 4 3 2 1

When I used the system, I felt transported to another time and place.

＊＊

11 8 8 0 1 0 0

Observing an ancient era felt like taking an actual trip to a new place.

＊＊

12 5 7 2 2 0 0

When I moved entities with the system, I felt as though I was actually there.

＊＊

12 4 7 3 2 0 0

I was not impacted emotionally by the events in the system. (-) 3 2 5 3 3 4 8

I experienced feelings as deeply in the system as I have in real life.

＊＊

16 7 4 1 0 0 0

When I used the system, I felt as though I was part of the ancient era.

＊＊

10 3 77 77 10 1 0

I reacted to the ancient environments and organisms in the system as though they were

real.

8 5 7 4 3 1 0

N=28 (-): reverse item

7:Strongly Agree, 6:Agree, 5:Agree Somewhat, 4:No Option, 3:Disagree Somewhat, 2:Disagree, 1:Strongly Disagree

＊＊

P<0.01

CSEDU 2016 - 8th International Conference on Computer Supported Education

184

Table 2 shows the results for the questionnaire

regarding the system’s ability to assist in learning. A

significant difference was observed for all the items

regarding the system’s learning assistance, in which

the neutral/negative group outnumbered the positive

group. This result implies that the participants

believed that the system helped them to learn about

ancient organisms and the eras in which they existed.

Table 3 shows the results for the questionnaire

regarding immersion. “I was not impacted

emotionally by events in the system” is a reverse item.

Therefore, we transferred the scores “strongly agree”

to “strongly disagree,” “agree” to “disagree,” and so

on. A significant difference was observed in six items

from this questionnaire, in which the neutral/negative

group outnumbered the positive group. A significant

difference was not detected in only one item, namely

“I was not impacted emotionally by the events in the

system.” These results imply that the participants

believed that the proposed system facilitated an

immersive paleoecological experience. However,

participants require more interactive and stimulating

events in order to be emotionally impacted by the

environment.

4 CONCLUSIONS

This paper summarized and evaluated an improved

version of the system developed by Yoshida et al. The

proposed system allows users to enrich their

understanding of paleontology, including extinct

animals and plants and their ecological environment.

We evaluated the proposed system in terms of its

ability to assist primary-school children learning

about such environments in museums, and in terms of

its potential to facilitate an immersed experience. The

results showed that the neutral/negative group

outnumbered the positive group. The results also

suggest that the proposed system helps users to learn

about ancient organisms and the eras in which they

existed and that it facilitated an immersive

paleoecological experience. In future work, we intend

to enhance the immersive experience of users in the

paleontological environment using a technique that

modifies the background according to the movements

of the learner.

ACKNOWLEDGEMENTS

This work was supported in part by Grants-in-Aid for

Scientific Research (B). The evaluation was

supported by the Museum of Nature and Human

Activities, Hyogo, Japan.

REFERENCES

Falk, J. H., Dierking, L. D., 2012. Museum Experience

Revisited. Left Coast Press. Walnut Creek California,

USA, 2

edition.

Stocklmayer, S. M., Rennie, L. J., Gilbert, J. K. 2010. The

roles of the provision of effective science education,

Studies in Science Education, 46(1), pp. 1-44,

Routledge Journals, Taylor & Francis Ltd.

Adachi, T., Goseki, M., Muratsu, K., Mizoguchi, H.,

Namatame, M., Sugimoto, M., Takeda, Y,. Fusako, K.,

Estuji, Y., Shigenori, I., 2013. Human SUGOROKU:

Full-body interaction system for students to learn

vegetation succession. Proceedings of the 12th

International Conference on Interaction Design and

Children. pp. 364-367, ACM.

Yoshida, R., Egusa, R., Saito, M., Namatame, M., Sugimoto,

M., Kusunoki, F., Etsuji, Y., Shigenori, I., Yoshiaki, T.,

Mizoguchi, H., 2015. BESIDE: Body Experience and

Sense of Immersion in Digital Paleontological

Environment. Proceedings of the 33rd Annual ACM

Conference Extended Abstracts on Human Factors in

Computing Systems. pp. 1283-1288. ACM.

Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio,

M., Moore, R., Kipman, A., Blake, A. 2011. Real-Time

Human Pose Recognition in Parts from a Single Depth

Image. Communications of the ACM. 56(1), pp. 116-

124, ACM.

Science Education Enhancement within a Museum using Computer-human Interaction Technology

185