Science Education Enhancement within a Museum using
Computer-human Interaction Technology
Haruya Tamaki
1
, Tsugunosuke Sakai
1
, Ryuichi Yoshida
1
, Ryohei Egusa
2,3
, Shigenori Inagaki
3
,
Etsuji Yamaguchi
3
, Fusako Kusunoki
4
, Miki Namatame
5
, Masanori Sugimoto
6
and Hiroshi Mizoguchi
1
1
Department of Mechanical Engineering, Tokyo University of Science, 2641 Yamazaki, Noda-shi, Chiba-ken, Japan
2
JSPS Research Fellow, Tokyo, Japan
3
Graduate School of Human Development and Environment, Kobe University, Hyogo, Japan
4
Department of Computing, Tama Art University, Tokyo, Japan
5
Tsukuba University of Technology, Ibaraki, Japan
6
Hokkaido University, Hokkaido, Japan
Keywords: Kinect Sensor, Museum, Immersive Learning Support System.
Abstract: We are developing an immersive learning support system for paleontological environments in museums. The
system measures the physical movement of the learner using a Kinect sensor, and provides a sense of
immersion in the paleontological environment by adapting the surroundings according to these movements.
As the first step toward realizing an immersive learning support system for museums, Yoshida et al. developed
and evaluated a prototype system. However, their system cannot learn about certain paleontological features
such as the names of extinct animals and their characteristics. Therefore, we developed an improved version
of this system that allows for an enriched knowledge of paleontological environments, focusing in particular
on extinct animals and plants and the ecological environment. Here, we evaluate the system’s learning
assistant and immersive features insofar as they are directed toward children in primary school. This paper
summarizes the current system and describes the evaluation results.
1 INTRODUCTION
Museums are important places for children to learn
about science (Falk and Dierking, 2012). They also
operate as centers for informal education in
connection with schools, and they enhance the
effectiveness of scientific education (Stocklmayer et
al., 2010). However, because the main learning
method within museums is the study of the specimens
on display and their explanations, there are few
chances for learners to observe or experience the
environment about which they are learning. In
particular, it is impossible to experience a
paleontological environment, which includes extinct
animals and plants and their ecological environment
(Adachi et al., 2013). It is difficult for children to
learn about such environments merely with fossils
and commentary. Overcoming this problem would
qualitatively improve scientific learning within
museums.
One solution to this problem would be some sort
of booth and video content that reproduced the
paleontological environment artificially. However,
issues pertaining to space and cost mean that most
museums cannot accommodate such an exhibit.
Hence, we are developing an immersive learning
system that will enable learners to explore a virtual
paleontological environment at any museum. This
“Body Experience and Sense of Immersion in a
Digital paleontological Environment” (or BESIDE)
system acquires information regarding the learner’s
movement using a Kinect sensor, and operates
according to this information. The system uses
multiple screens spread across the learner’s entire
field of vision. By being projected into this virtual
space, the learner can adopt physical movements as
observational actions. We expect that this will
engender a sense of immersion in the virtual space.
Because BESIDE comprises only a commercial
image sensor, projector, and control PC, we can
provide a low-cost immersive learning experience
within a small space.
As the first step toward realizing an immersive
learning support system for museums, Yoshida et al.
Tamaki, H., Sakai, T., Yoshida, R., Egusa, R., Inagaki, S., Yamaguchi, E., Kusunoki, F., Namatame, M., Sugimoto, M. and Mizoguchi, H.
Science Education Enhancement within a Museum using Computer-human Interaction Technology.
In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016) - Volume 2, pages 181-185
ISBN: 978-989-758-179-3
Copyright
c
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
181
developed and evaluated a prototype system (Yoshida
et al., 2015).
Their system allows the learner to experience the
paleontological environment. By being projected in
such a paleontological environment, the learner can
adopt physical movements as observational actions.
This engenders a sense of immersion in the
paleontological environment. However, their system
is limited insofar as users cannot learn about these
paleontological environments, including the extinct
animals and plants and their ecological environment.
Because of this limitation, we developed an improved
version of the system developed by Yoshida et al.
With our proposed system, learners can develop a
richer knowledge of paleontological environments,
including extinct animals and plants and the
ecological environment.
In this paper, we summarize our improved version
of the system developed by Yoshida et al., and we
describe an evaluation of its learning support and
immersion.
2 IMMERSIVE LEARNING
SUPPORT SYSTEM
2.1 BESIDE
We are developing “BESIDE” as an immersive
learning system that enables learners to explore a
virtual paleontological environment at any museum.
BESIDE consists of various sensors and digital
learning content. The sensors measure the learners’
location, pose, and actions, and the learning content
is then controlled according to these measurements.
Figure 1 illustrates the concept of BESIDE.
Figure 1: Concept of BESIDE.
Learners walk around a space containing a screen
that displays a virtual environment, and observable
objects such as animals move in synchronization with
them. In this way, learners feel immersed in the
paleontological environment. Synchronizing the
movement of the paleontological animals with that of
the learner makes it possible to consider the animals
“real” in some sense, rather than imaginary.
Furthermore, by introducing near-real activities,
such as “approaching the observable object” or
“diving into water,” the level of interest and learning
effects are more enhanced than they would be by
merely attending a typical exhibition or watching a
video.
2.2 Current System
We are developing a system that includes a figure of
the learner in a virtual environment displayed on the
screen, allowing the learner to experience the
paleontological environment and learn about extinct
animals and plants and the ecological environment.
Figure 2 shows the setup of the current system.
Figure 2: Setup of the current system.
In the system, the learner first stands in front of
the screen, and a camera image of the learner is
displayed with the background removed. Learners
can change the background by moving their hands
and interacting with the content. The system then
allows learners to select animals appropriate for
different geologic eras, and these are then displayed
on the screen. Furthermore, the animal’s
characteristics are displayed in the screen. Learners
use their hands to select the animal’s name and then
interact with the content.
The system must be able to acquire real-time data
regarding the learner’s location and movements. Thus,
we utilize Microsoft’s Kinect sensor, a range-image
sensor originally developed as a home videogame
device. Although it is inexpensive, the sensor can
record sophisticated measurements regarding the
user’s location (Shotton, J., 2011). Additionally, this
sensor can recognize humans and the human skeleton
using the library in Kinect’s software development kit
for Windows. Kinect can measure the location of
human body parts such as hands and legs, and it can
CSEDU 2016 - 8th International Conference on Computer Supported Education
182
identify the user’s pose or status with this function
and the location information. We use these functions
to recognize humans and detect the human skeleton.
The system reproduces a paleontological
environment by placing a suitable animal from one of
three geologic eras (viz., Paleozoic, Mesozoic, and
Cenozoic) into the display. The animals appropriate
to each era are listed in Table 1. We also prepared
three kinds of fossils that are typical of these
animals—i.e., fossils that can be found at the museum.
Furthermore, we prepared animals’ names and
characteristics. With this system, information
regarding the frames and depth are measured with the
Kinect sensor. The measurements are sent to the PC,
and various images are displayed on the screen. The
system has the following functions:
(a) Displays images based on sensor information;
(b) Operates using the learner’s body motion;
(c) Enables observations of animals as GIF
animations.
(d) Teaches animal names and characteristics
Table 1: Details of animals.
Paleozoic Mesozoic Cenozoic
Elrathiakingi
(Trilobite)
Perisphinctina
(Ammonoidea)
Mammoth
Orthoceras Triceratops Merycoidodon
Crinoidea Mosasaurus Shark
Function (a) displays the user on the screen by
recognizing the user’s outline. Accordingly, the user
is placed into the paleontological environment.
Function (b) allows the system to operate according
to the user’s hand movements. The learner can click
a button by pushing his/her hand toward the Kinect
sensor. The learner then chooses one of the three
geologic eras, and the screen is replaced with an
image appropriate to that era. Icons of the animals are
displayed at the bottom the screen. If the learner
selects an animal from that geologic era, the animal is
displayed on the screen. If the selected animal is from
a different era, a warning sounds and the learner is
asked to choose again. This enables the system to
teach the learner about which animals lived in each
era, in addition to providing a sense of immersion.
Function (c) enables the displayed animal to move
around the screen as a GIF animation. By means of
this, learners can observe extinct animals as though
they were real. Function (d) displays animals’ names
at the bottom the screen. If the learner selects the
correct name of the animal, its characteristics are
displayed on the screen. If the selected animal is from
the wrong era, a warning sounds and the learner is
asked to choose again. By means of this, the user
learns about paleontological environments, including
extinct animals and plants and their ecological
environment. Figure 3 shows a learner using the
current systems’ functions.
Function (a)
Function (b)
Function (c)
Function (d)
Figure 3: Functions of the current system.
Figure 4: Using the current system.
Science Education Enhancement within a Museum using Computer-human Interaction Technology
183
3 EVALUATION
3.1 Method
Participants: The participants comprised 27 students
(age: 10–11 years) from an elementary school in
Kobe, Japan (12 boys and 15 girls).
Process: The participants were organized into groups
of 3–4, and were instructed to observe selected fossils
on display at the museum. These fossils were from
organisms available in the prototype system.
Participants used the system, one by one. The
participants were able to select the geologic era
relevant to the fossils they studied—viz., Paleozoic,
Mesozoic, and Cenozoic—and the paleoecology of
the era was then simulated. We prepared three
silhouette quizzes and three quizzes regarding animal
names for each geologic era. Each of the participants
spent approximately five minutes interacting with the
system. Finally, the participants evaluated the system.
Evaluation: A questionnaire was used to evaluate two
features of the system: learning assistance and
immersion. The questionnaire consisted of 12 items.
The learning assistance provided by the system was
evaluated with five questions regarding changes in
the understanding and interests of the participants
after experiencing the paleoecological simulation.
Immersion was evaluated with seven questions
(based on Player Experience of Need Satisfaction
Physical/Emotional/Narrative Presence Scale)
regarding the effectiveness of the system at enhancing
the immersive experience of the paleoecological
simulation and the learner’s projection into the
scenery. For each question, participants answered,
“strongly agree,” “agree,” “agree somewhat,”
“neutral,” “disagree,” “disagree somewhat,” or
“strongly disagree.”
Date of Experiment: July 25, 2015
Location: Natural Science Museum, Hyogo
Prefecture.
3.2 Results
The average scores for each quiz were as follows: the
silhouette quizzes resulted in an average of 2.68, and
the quizzes regarding animal names resulted in an
average of 2.82.
Data obtained from the questionnaire was divided
into two groups—positive evaluations (“strongly
agree,” “agree,” and “agree somewhat”), and neutral or
negative evaluations (“neutral,” “disagree,” “disagree
somewhat,” and “strongly disagree”)—and the results
for the two groups were totaled. The difference
between the positive and neutral/negative evaluations
was computed using Fisher’s Exact Test (1 x 2).
Table 2: Results of evaluation about learning assistance.
Item 7 6 5 4 3 2 1
When I accomplished all the quizzes in the system, I experienced genuine pride.
**
17 5 6 0 0 0 0
When I read the explanations of the ancient organisms provided by the system, I believed
that they were effective at improving the learning experience at the museum.
**
15 6 5 1 0 1 0
Using body motion in order to complete the quizzes was fun.
**
20 5 3 0 0 0 0
The silhouette quizzes were effective at improving the learning experience at the museum.
**
19 6 2 1 0 0 0
The quizzes regarding animal names were effective at improving the learning experience at
the museum.
**
21 4 2 1 0 0 0
N=28
7:Strongly Agree, 6:Agree, 5:Agree Somewhat, 4:No Option, 3:Disagree Somewhat, 2:Disagree, 1:Strongly Disagree
**
P<0.01
Table 3: Results from the Evaluation of the Proposed System’s Immersion.
Item 7 6 5 4 3 2 1
When I used the system, I felt transported to another time and place.
**
11 8 8 0 1 0 0
Observing an ancient era felt like taking an actual trip to a new place.
**
12 5 7 2 2 0 0
When I moved entities with the system, I felt as though I was actually there.
**
12 4 7 3 2 0 0
I was not impacted emotionally by the events in the system. (-) 3 2 5 3 3 4 8
I experienced feelings as deeply in the system as I have in real life.
**
16 7 4 1 0 0 0
When I used the system, I felt as though I was part of the ancient era.
**
10 3 77 77 10 1 0
I reacted to the ancient environments and organisms in the system as though they were
real.
8 5 7 4 3 1 0
N=28 (-): reverse item
7:Strongly Agree, 6:Agree, 5:Agree Somewhat, 4:No Option, 3:Disagree Somewhat, 2:Disagree, 1:Strongly Disagree
**
P<0.01
CSEDU 2016 - 8th International Conference on Computer Supported Education
184
Table 2 shows the results for the questionnaire
regarding the system’s ability to assist in learning. A
significant difference was observed for all the items
regarding the system’s learning assistance, in which
the neutral/negative group outnumbered the positive
group. This result implies that the participants
believed that the system helped them to learn about
ancient organisms and the eras in which they existed.
Table 3 shows the results for the questionnaire
regarding immersion. “I was not impacted
emotionally by events in the system” is a reverse item.
Therefore, we transferred the scores “strongly agree”
to “strongly disagree,” “agree” to “disagree,” and so
on. A significant difference was observed in six items
from this questionnaire, in which the neutral/negative
group outnumbered the positive group. A significant
difference was not detected in only one item, namely
“I was not impacted emotionally by the events in the
system.” These results imply that the participants
believed that the proposed system facilitated an
immersive paleoecological experience. However,
participants require more interactive and stimulating
events in order to be emotionally impacted by the
environment.
4 CONCLUSIONS
This paper summarized and evaluated an improved
version of the system developed by Yoshida et al. The
proposed system allows users to enrich their
understanding of paleontology, including extinct
animals and plants and their ecological environment.
We evaluated the proposed system in terms of its
ability to assist primary-school children learning
about such environments in museums, and in terms of
its potential to facilitate an immersed experience. The
results showed that the neutral/negative group
outnumbered the positive group. The results also
suggest that the proposed system helps users to learn
about ancient organisms and the eras in which they
existed and that it facilitated an immersive
paleoecological experience. In future work, we intend
to enhance the immersive experience of users in the
paleontological environment using a technique that
modifies the background according to the movements
of the learner.
ACKNOWLEDGEMENTS
This work was supported in part by Grants-in-Aid for
Scientific Research (B). The evaluation was
supported by the Museum of Nature and Human
Activities, Hyogo, Japan.
REFERENCES
Falk, J. H., Dierking, L. D., 2012. Museum Experience
Revisited. Left Coast Press. Walnut Creek California,
USA, 2
nd
edition.
Stocklmayer, S. M., Rennie, L. J., Gilbert, J. K. 2010. The
roles of the provision of effective science education,
Studies in Science Education, 46(1), pp. 1-44,
Routledge Journals, Taylor & Francis Ltd.
Adachi, T., Goseki, M., Muratsu, K., Mizoguchi, H.,
Namatame, M., Sugimoto, M., Takeda, Y,. Fusako, K.,
Estuji, Y., Shigenori, I., 2013. Human SUGOROKU:
Full-body interaction system for students to learn
vegetation succession. Proceedings of the 12th
International Conference on Interaction Design and
Children. pp. 364-367, ACM.
Yoshida, R., Egusa, R., Saito, M., Namatame, M., Sugimoto,
M., Kusunoki, F., Etsuji, Y., Shigenori, I., Yoshiaki, T.,
Mizoguchi, H., 2015. BESIDE: Body Experience and
Sense of Immersion in Digital Paleontological
Environment. Proceedings of the 33rd Annual ACM
Conference Extended Abstracts on Human Factors in
Computing Systems. pp. 1283-1288. ACM.
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio,
M., Moore, R., Kipman, A., Blake, A. 2011. Real-Time
Human Pose Recognition in Parts from a Single Depth
Image. Communications of the ACM. 56(1), pp. 116-
124, ACM.
Science Education Enhancement within a Museum using Computer-human Interaction Technology
185