Sound Woods: An Interactive Game-Based Learning Design for Inclusive

Play Between Sighted and Visually Impaired Users

Chenyi Dai

, Kojiro Hirose

, Makoto Kobayashi

, Shigenori Inagaki

and Fusako Kusunoki

Information Design Department, Tama Art University, Hachioji, Tokyo, Japan

National Museum of Ethnology, Osaka, Japan

Tsukuba University of Technology, Ibaraki, Japan

Department of Education, Rikkyo University, Toyoshima, Tokyo, Japan

Keywords:

Visual Impairment, Game-Based Learning, Orientation & Mobility Training, Virtual Learning Environment,

Auditory Feedback, Tangible User Interface, Game Design, Location-Based Game.

Abstract:

In traditional spatial learning for the visually impaired, preliminary understanding of spatial concepts is typically

formed through oral descriptions by instructors prior to actual environmental engagement. However, this method

is restrictive. This study encourages users to recreate and experience environmental spatial sounds within a

realm by ”Sound woods” using Orientation & Mobility Training. This enables users to engage in auditory

training, focusing on ”listening and positioning” to enhance environmental spatial cognition. For the visually

impaired, who often have limited opportunities to interact with sighted individuals in their daily lives, video

games offer potential avenue for social engagement. Traditional video games primarily rely on visual interaction

through screens, posing challenges to visually impaired individuals owing to their limited eyesight. We designed

a tangible video game that provides game-based learning, fostering communication between visually impaired

and sighted individuals. Additionally, it provides sighted individuals the opportunity to experience space

without sight, increasing empathy for the visually impaired, likely alleviating discrimination to some extent.

This study underwent three rounds of testing, involving visually impaired individuals, individuals related to

the visually impaired, and approximately 20 sighted participants in the workshop tests, incorporating their

experiences and insights through interviews.

1 INTRODUCTION

The population of visually impaired individuals across

countries and regions is signiﬁcant. In Japan, approxi-

mately 1.64 million visually impaired people reported

in 2007, a number expected to reach 2 million by 2030

(Yamada et al., 2010).Unfortunately, for these individ-

uals, access to everyday life and leisure activities is

often limited. For instance, children with visual im-

pairments may interact with their environment less,

which limits the educational beneﬁts school activities

accords to them (Freeman et al., 2017). Navigating the

surroundings is a signiﬁcant challenge for the visually

impaired (Regal et al., 2018), necessitating consistent

participation in spatial training and learning. Tradi-

tional methods of spatial training and learning often

rely on oral descriptions provided by an instructor prior

to actual engagement with the environment, but this

approach has certain limitations.Therefore, this study

proposes a shift toward entertainment-based and ubiq-

uitous learning. It aims to engage users through the

exploration of sound locations in a customized virtual

world, utilizing gamiﬁed learning for Orientation &

Mobility Training (O&M).

Studies exhibit that video games are an integral

part of children and youth culture. Furthermore, these

games are increasingly utilized by a growing segment

of the population, particularly young adults (average

age 25, with women accounting for 40%), and the pro-

portion of players in other age groups is also rising

(Archambault et al., 2007). However, current video

games generally lack standardized methods for convey-

ing on-screen information through non-visual means.

Consequently, individuals who cannot utilize normal

graphical interfaces, particularly those who are com-

pletely blind or have severe visual impairments (visual

acuity less than 0.05), are restricted or entirely un-

able to access this signiﬁcant aspect of youth culture

(Buaud et al., 2002).

Game-based learning is deﬁned as the use of games

as learning environments where educational content

is integrated into games, facilitating the learning pro-

cess through engagement and interaction (Plass et al.,

2015).In this study, combined with O&M Training,

Dai, C., Hirose, K., Kobayashi, M., Inagaki, S. and Kusunoki, F.

Sound Woods: An Interactive Game-Based Learning Design for Inclusive Play Between Sighted and Visually Impaired Users.

DOI: 10.5220/0012695700003693

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024) - Volume 2, pages 693-700

ISBN: 978-989-758-697-2; ISSN: 2184-5026

693

Figure 1: Sound Woods.

we developed a tangible video game ”Sound Woods.”

The ﬁeld of game design for the visually impaired has

seen signiﬁcant advancements in recent research. An

exemplary design is the Binaural Navigation Game,

a 3D audio game designed to enhance the abilities

of both visually impaired and sighted players, focus-

ing on training the visually impaired (Balan et al.,

2015).Another example is ”Em Busca do Santo Grau

– Accessible Version,” a 2D adventure game that is

designed to be multidisciplinary and customizable for

various educational levels (Neto et al., 2020). These

instances demonstrate the potential of game-base learn-

ing design for the visually impaired.

Focusing on the design problems exhibited in Table

1 and considering the solutions provided, I concentrate

on designing a game that is inherently enjoyable with-

out the need for vision. Such games exist in many

different forms, utilizing unique methods to engage

players by incorporating other senses. For example,

conversational games, tactile board games, and audio

games, among others. This project ultimately adopts

a tactile tabletop electronic game format, combined

with audio feedback for design.

Before visually impaired individuals undergo real-

world training for environmental spatial cognition,

they often form preliminary impressions based on ver-

bal descriptions from instructors. This traditional ap-

proach is somewhat limited. One of the major chal-

lenges faced by individuals with visual disabilities is

being aware of what their current position is in differ-

ent surroundings (Fa

c¸

anha et al., 2020). This game

will recreate environmental spatial sounds in a virtual

world, enabling players to train their auditory skills,

particularly ”sound localization,” to learn about en-

vironmental spatial cognition. Humans can identify

the direction and location of sounds utilizing differ-

ences in the timing and amplitude of sounds reaching

their ears. By speciﬁcally, by discerning which ear the

sound reaches ﬁrst and the difference in sound intensity

between the ears, the source can be perceived. This

study utilizes stereophonic technology to implement

Table 1: Design problems and solutions.

Problem Solution

For sighted individuals,

playing games designed

for the visually impaired

may seem unnecessary

in daily life, making it

challenging to engage in

these games intentionally.

Additionally, there are

concerns about fairness

in these games, as dif-

fering visual abilities can

make playing together

potentially unfair for vi-

sually impaired partici-

pants.

It would be beneﬁcial

to reference games

that can be enjoyed

without visual ele-

ments. The design

philosophy should

shift, not forcing

sighted individuals to

experience an unseen

environment, but

rather transforming

the absence of vision

into an engaging

challenge and point

of interest in the

game.

For visually impaired

individuals, recognizing

the position of characters

or obstacles and deter-

mining if inputs are cor-

rect can be difﬁcult.

Use various feedback

that incorporates

sound. Aim to

provide a more

accurate sense of

distance and direction

through headphones

by stereophonizing

the sounds within the

relevant scene of the

game.

3D spatial environment sounds, as it can clearly convey

the location of sounds. Incorporating this technology

into the game is expected to support the training and

improvement of players’ sound localization abilities.

CSEDU 2024 - 16th International Conference on Computer Supported Education

694

Figure 2: Game play.

2 IMPLEMENTATION OF THE

GAME

2.1 Game Description

In this study, the developed “Sound Woods”(Figure 1)

introduces an innovative interactive spacial learning

experience, where both visually impaired and sighted

individuals can participate together. The concept of the

game revolves around “planting” sounds in a virtual

space as seeds, where players are tasked with locating

these “seeds of sound” through active listening.

In detail, at the beginning of the game, a player

plants speciﬁc sounds into the virtual space. The op-

posing player, wearing headphones, distinguishes and

searches for these sounds’ positions within the space

utilizing stereo audio. Players then switch roles for the

next round, and the player who identiﬁes the sounds

quicker wins.

By eliminating visual elements and focusing solely

on auditory ones, the game’s design offers a fair and

stimulating experience to both the visually impaired

and sighted players. It aims to deepen the understand-

ing and empathy for the world of those with visual

impairments.

2.2 Game Play

The gameplay is divided into two main parts (Figure 2).

The ﬁrst part termed ”planting sounds,” involves Player

1 planting sound seeds in a physical box, aligning with

speciﬁc locations in the virtual world. The second

part, ”searching for sounds,” has Player 2, wearing

headphones and utilizing the game’s 3D audio tech-

nology to perceive the distance and direction of the

sound.The objective is for Player 2 to locate the sound

seeds planted by Player 1 solely through auditory cues,

eliminating the reliance on sight.

2.3 Game System

The ”Sound Woods” game system (Figure 3) utilizes

the ESP32 to enable connection between interactive

physical elements and the computer. The system em-

ploys magnetic sensors to receive information about

where the sound seeds, crafted from clay and magnets,

are placed. This enables players to participate directly

in the game by physically planting seeds, with these

actions reﬂected in the virtual world of the game.

•

Structure of the Game System. The game system

comprises two main components; the hardware

interface and the software platform. The hard-

ware interface utilizes the ESP32 and magnetic

sensors to collect data on the placement of sound

seeds and sends this information to the software

platform. The software platform, powered by the

Unity engine, reﬂects this data in the virtual world,

arranging and playing back the relevant sounds

accordingly.

•

About ESP32. The ESP32 is a low-cost, low-power

microcontroller chipset which is widely utilized

in Internet of Things (IoT) projects and embedded

system applications.

•

Communication Principle between ESP32 or Ar-

duino and Unity: The communication between

ESP32 or Arduino and Unity utilizes serial commu-

nication protocols. This reliable method enables

efﬁcient data transmission between the hardware

devices and computers.

•

Data Transmission. When a player physically

plants a sound seed, the magnetic sensor detects

this action and sends the related data to the ESP32

or Arduino. Subsequently, the ESP32 or Arduino

transmits this data to the computer through serial

communication.

•

Data Reception and Processing. The computer

receives data through serial communication, and

the Unity engine analyzes this data to perform the

Sound Woods: An Interactive Game-Based Learning Design for Inclusive Play Between Sighted and Visually Impaired Users

695

Figure 3: Game system.

Figure 4: Communication process.

placement and playback of relevant sound seeds in

the virtual world.

•

Feedback Provision. The Unity engine provides

visual and auditory feedback based on the place-

ment and playback of sounds, assisting players to

understand how their actions are reﬂected within

the game.

2.4 Communication Process

The communication process (Figure 4) occurs in real-

time, offering immediate feedback to the player, ul-

timately enhancing the gameplay experience. Addi-

tionally, this process improves the ability of visually

impaired players to participate in the game through

physical actions and interact with the virtual world

through sound.

3 EXPERIMENT

This study collected a rich array of opinions through

three rounds of testing, involving meetings with visu-

ally impaired related researchers and a workshop that

involved 18 participants.

Figure 5: User test with Professor Hirose.

3.1 User Test with Professor Hirose

3.1.1 Method

Professor Hirose is a visually impaired cultural anthro-

pologist who is a professor at the National Museum of

Ethnology in Japan. He contributes to foundational re-

search in anthropology. His research spans the history

of religion in Japan, folklore, cultural theory of disabil-

ity, and tactile culture. The user test for this study was

conducted with Professor Hirose after an oral explana-

tion of how the game is played, followed by capturing

and analyzing his experience and opinions (Figure 5).

3.1.2 Result

With regards to the result and feedback, regarding

the gameplay, Professor Hirose understood the rules

clearly and completed a sequence of the game within

one minute. He suggested that a clear indication is

needed when a sound is found. Additionally, regarding

the signiﬁcance and application scenarios of the game,

he reﬂected on his ﬁrst experience at Hachioji Station,

noting that starting from the exit and ascending the

stairs, various sounds could be heard, and it was a

good choice to calculate distances and scenarios by

listening to these sounds. He pointed out that prior

to actual activities, the game could provide training

opportunities for visually impaired individuals.

3.2 Workshop at the National Museum

of Ethnology

3.2.1 Method

In the workshop conducted at the National Museum

of Ethnology in Osaka (Figure 6), approximately 20

CSEDU 2024 - 16th International Conference on Computer Supported Education

696

Table 2: Overview of participants.

Number Sex Average Age

18 F(10) M(8) 39.5

Figure 6: Workshop at the National Museum of Ethnology.

sighted individuals participated, and 18 valid question-

naires were collected (Table 2). During the workshop,

researchers explained the game’s rules verbally with a

demonstration that was carried out. participants were

then paired up to take turns simulating visual impair-

ment during the test.

3.2.2 Result

As for the results, the survey data (Tables 3, 4) indi-

cates that the satisfaction with the game’s feedback

method was moderate, and the participants’ under-

standing of the game rules was good. The evaluation

of the sound design was also moderate, with sugges-

tions made for the addition of everyday sounds like

the crunch of potato chips, sneezing, animal calls, and

musical instruments.

3.3 Meeting with Professor Kobayashi

from Tsukuba University of

Technology

3.3.1 Method

During the meeting with Professor Kobayashi from

Tsukuba University of Technology, which caters to stu-

dents with visual and hearing impairments, he shared

numerous suggestions and related research insights.

These contributions stemmed from his experience with

”Sound Woods,” involving the participation of a visu-

ally impaired student.

3.3.2 Result

Professor Kobayashi’s feedback and advice included

the following proposals:

•

When utilizing only a keyboard, it’s not possible

to know the absolute position (which row and col-

umn). Utilzing a slider or other tangible objects

could be a better solution. Alternatively, inform

the player of their position during the response.

Additionally, each cell could be changed into a

physical button, with the player identifying the

sound by pressing the button.

•

Considering the integration the game with exist-

ing ones, Professor Kobayashi demonstrated a Go

game designed speciﬁcally for the visually im-

paired. This demonstration exhibits a clever design

for board games tailored for individuals with visual

impairments.

4 CONCLUSIONS AND FUTURE

WORK

This study describes the development of a tangible

video game called the ”Sound Woods,” designed to

bridge the experiential gap between visually impaired

and sighted players. For visually impaired individuals,

we developed a game that employs auditory learning

through sound localization, which is crucial for envi-

ronmental spatial cognition.

The effectiveness of the system was evaluated uti-

lizing a questionnaire consisting of detailed queries

on usability, learning outcomes, and social interaction.

Positive responses indicated that the game succeeded

in its educational intent, as participants demonstrated

improved skills in auditory localization and enhanced

environmental awareness.

The results of user tests revealed that the game has

the potential to offer social, educational, and technical

beneﬁts. While still in its early prototype phase, the

game aims to create an inclusive gaming environment

for both visually impaired and sighted individuals, pro-

moting equality and accessibility. It seeks to reduce

discrimination and support auditory training. Addi-

tionally, it broadens the scope of game interactions

and incorporates innovative gaming interfaces, thereby

enriching the gaming experience.

Participants reported that the game was engaging

and showed potential as a learning tool, highlighting its

role in promoting unique social interactions. The game

was effective in providing an inclusive experience, as

sighted players also reported a deeper understanding

and empathy for the visually impaired. The results

of the questionnaire also suggest that there are many

areas for improvement in the game, such as the need

for better differentiation in game difﬁculty levels, and

the introduction of a wider variety of sounds from

Sound Woods: An Interactive Game-Based Learning Design for Inclusive Play Between Sighted and Visually Impaired Users

697

Table 3: Results of Questionnaire Part 1.

Question 1 2 3 4 5 Num AVR.

Q1: Are you satisﬁed with the feedback and hints provided regarding sounds

within the game?

0 6 5 3 4 18 3.28

Q2: Was the mechanism of planting sounds and using them for navigation within

the game easy to understand?

0 1 5 6 6 18 3.94

Q3: After closing your eyes, did you feel the sound effects in the game sufﬁ-

ciently conveyed the environment and scenes?

0 3 10 2 3 18 3.28

Q4: Do you think the difﬁculty level in the game is appropriate? 0 5 3 6 4 18 3.50

Q5: Are you satisﬁed with the variety and timbre of sounds in the game? 0 0 4 8 6 18 4.11

Q6:Did you think ”Sound Woods” would help to discover new methods of

communication with visually impaired individuals?

0 3 8 5 2 18 3.33

Q7: This game is designed to be played without relying on sight, are you satisﬁed

with your experience so far?

0 1 3 7 4 15 3.93

1-5: score. 1: Not at all, 2: Not much, 3: Neutral, 4: Somewhat, 5: Very

Table 4: Results of Questionnaire Part 2.

Q10:Do you see potential for this game in the ﬁelds of education, entertainment, or social interaction? Please share

any other comments, ideas, or suggestions you might have.

User Answer

USER1 I like the idea of planting sounds. Looking forward to improvements on explaining the rules better.

USER2

The idea of planting sounds was innovative, like sounds of eating chips or musical instruments.

Making magnets in the shape of sprouts seems fun.

USER3

It was challenging to differentiate among the three types of sounds. Completely different sounds

would be easier to distinguish.

USER4 There is potential for application.

USER5

Moving randomly seemed to end the game; more digital movement could make the game more

engaging.

USER6 The search for sound sources reminded me of TREASURE IN DARK at Teikyo University.

USER7 -

USER8 Enjoying the game together was fun, and the controls were user-friendly.

USER9 High difﬁculty for sighted people was interesting; adding animal sounds might be good.

USER10 -

USER11 -

USER12

Adjusting the game level by choosing the sound volume with more or fewer seeds could be interesting.

Other sounds, like those in a station, could be engaging.

USER13 The game was enjoyable once the rules were clear.

USER14 -

USER15 -

USER16 -

USER17 Increasing the variety of sounds would be better.

USER18 -

-: no answer.

CSEDU 2024 - 16th International Conference on Computer Supported Education

698

everyday life in the virtual game world to enhance

playability.

Professor Hirose, participating as a visually im-

paired individual, afﬁrmed the game’s extensive appli-

cability, particularly its utility in spatial training for

the visually impaired. Prior to navigating unfamiliar

environments, such as new train stations, users could

leverage the game’s simulation of spatial audio cues in

a virtual setting for preparatory learning. This feature

underscores the game’s potential as a practical tool for

real-world orientation and mobility training for visu-

ally impaired users, demonstrating the game’s capacity

to extend beyond entertainment and into practical ap-

plication.

The results of this study also suggest that integrat-

ing the learning component into gaming can enhance

the accessibility and educational value of games for

the visually impaired. These ﬁndings establish the po-

tential of tactile video games as a medium for inclusive

education and entertainment.

The aim of this study is to develop accessible game-

based learning experiences that serve as effective plat-

forms while promoting social inclusion. By imple-

menting such a system, we aspire to enhance the eq-

uity of the gaming landscape, providing opportunities

for entertainment, education, and meaningful social

interaction to both visually impaired and sighted indi-

viduals.

In future work, ”Sound Woods” will focus on fur-

ther expanding its capabilities as a game-based learn-

ing tool for spatial orientation and mobility training.

Key developments will include:

•

Enhanced Sound Design. To better replicate real

environments and therefore improve spatial cogni-

tion learning, the game will offer a more extensive

selection of sounds for players to place in the vir-

tual space. This could involve providing a broader

range of environmental sounds or increasing the

size of the playable area by adding more sensors.

Such enhancements will allow for a more realistic

and immersive auditory experience, aiding in the

development of spatial awareness and navigation

skills.

•

Advanced Tactile Interfaces. The integration of

advanced tactile interfaces will aim to provide a

more engaging and interactive experience. By im-

plementing haptic feedback systems, players will

be able to experience different textures and vibra-

tions, which will enhance their understanding of

the virtual environment and add a new dimension

to the learning experience.

•

Customizable Difﬁculty Levels. Recognizing the

varied abilities and experiences of its players,

”Sound Woods” will include customizable difﬁ-

culty settings. This feature will ensure that the

game remains appropriately challenging and edu-

cational for all players, adapting to their speciﬁc

needs and skill levels.

•

Educational Content Integration. The game will

integrate real-world spatial environments that re-

quire navigational learning, or combine story-

telling, such as recreating the space inside a castle

from a movie. This approach will not only make

the learning experience more engaging but also

provide cultural and interesting context, making

”Sound Woods” an innovative tool for spacial learn-

ing.

ACKNOWLEDGEMENTS

This work was supported by JSPS KAKENHI Grant

Number 22H00078, 22H01068. The evaluation work-

shop was supported by the National Museum of Eth-

nology.

REFERENCES

Archambault, D., Ossmann, R., Gaudy, T., and Miesenberger,

K. (2007). Computer games and visually impaired

people. Upgrade, 8(2):43–53.

Balan, O., Moldoveanu, A., Moldoveanu, F., and Butean,

A. (2015). Developing a navigational 3d audio game

with hierarchical levels of difﬁculty for the visually

impaired players. In RoCHI, pages 49–54.

Buaud, A., Svensson, H., Archambault, D., and Burger,

D. (2002). Multimedia games for visually impaired

children. In Computers Helping People with Special

Needs, pages 173–180. Springer.

c¸

anha, A. R., Darin, T., Viana, W., and S

anchez, J. (2020).

O&m indoor virtual environments for people who are

blind: A systematic literature review. ACM Transac-

tions on Accessible Computing, 13(2).

Freeman, E., Wilson, G., Brewster, S., Baud-Bovy, G., Mag-

nusson, C., and Caltenco, H. (2017). Audible beacons

and wearables in schools: Helping young visually im-

paired children play and move independently. In Pro-

ceedings of the 2017 CHI Conference on Human Fac-

tors in Computing Systems, CHI ’17, page 4146–4157,

New York, NY, USA. Association for Computing Ma-

chinery.

Neto, L. V., Fontoura Junior, P. H., Bordini, R. A., Otsuka,

J. L., and Beder, D. M. (2020). Design and implemen-

tation of an educational game considering issues for

visually impaired people inclusion. Smart Learning

Environments, 7(1):4.

Plass, J. L., Homer, B. D., and Kinzer, C. K. (2015). Foun-

dations of game-based learning. Educational psychol-

ogist, 50(4):258–283.

Sound Woods: An Interactive Game-Based Learning Design for Inclusive Play Between Sighted and Visually Impaired Users

699

Regal, G., Mattheiss, E., Sellitsch, D., and Tscheligi, M.

(2018). Mobile location-based games to support ori-

entation & mobility training for visually impaired stu-

dents. In Proceedings of the 20th International Con-

ference on Human-Computer Interaction with Mobile

Devices and Services, MobileHCI ’18, pages 1–12,

New York, NY, USA. Association for Computing Ma-

chinery.

Yamada, M., Hiratsuka, Y., Roberts, C. B., Pezzullo, M. L.,

Yates, K., Takano, S., Miyake, K., and Taylor, H. R.

(2010). Prevalence of visual impairment in the adult

japanese population by cause and severity and future

projections. Ophthalmic Epidemiology, 17(1):50–57.

CSEDU 2024 - 16th International Conference on Computer Supported Education

700