Basic Research on Multisensory Methods for Teaching Onomatopoeia to

the Hearing-impaired

Broadening the Experience of Sound

Miki Namatame

, Fusako Kusunoki

and Shigenori Inagaki

Department of Industrial Information, Tsukuba University of Technology, Amakubo, Tsukuba, Ibaraki 305-8520, Japan

Department of Information Design, Tama Art University, Yarimizu, Hachioji, Tokyo 192-0394, Japan

Faculty of Human Development, Kobe University, Tsurukabuto, Nada-ku, Kobe 657-8501, Japan

Keywords:

Hearing-impaired, Science Lesson, Cicada, Vibration, Onomatopoeia.

Abstract:

It is difﬁcult for hearing-impaired individuals to learn animal sounds and the onomatopoeia that depict them

because they have limited or no access to auditory information. To attempt to solve this problem, the author

designed a science lesson to help hearing-impaired students learn cicada songs. This lesson used multiple

media including text, images, sounds, sound waveforms, onomatopoeia and vibrations to stimulate the senses

of sight, hearing and touch. An experiment was conducted on 26 hearing-impaired students, who were split

into two groups. One of the groups was provided with vibrations as part of the lesson and the other was not.

Pre- and post-tests on the names of cicadas and their songs and a lesson evaluation survey were conducted to

assess the lesson’s effectiveness. Although results showed no signiﬁcant difference in learning between the

two groups, students who were able to discriminate among the different vibrations reported that they found

the lesson enjoyable and that it was a useful way to learn science.

1 INTRODUCTION

The Japanese language is said to use from three to

ﬁve times as many onomatopoeic words as western

languages or Chinese (Yamaguchi, 2003). The one

dictionary of Japanese mimetics lists 4500 entries

(Ono, 2007). Educational research has suggested that

onomatopoeia, that is, words which mimic animate

sounds (giseigo), inanimate sounds (giongo) or are

symbolic of states (gitaigo), of emotions (gijougo)

or of actions (giyougo), are easy for students to un-

derstand and enhance their comprehension and ima-

gination (Miyazaki and Tomimatsu, 2009), that high

iconicity between the sound and the referent enables

listeners to accurately generalize the meaning of the

word and to make an immediate connection with the

object being symbolized, and, that sound symbolic

words referring to states, emotions or actions facili-

tate the learning of verbs as children are acquiring vo-

cabulary(Imai and Kita, 2014). So learners of the Ja-

panese language must master onomatopoeia to make

their Japanese more descriptive and expressive(Yusuf

and Watanabe, 2008).

In this way, onomatopoeia plays an important role

in word acquisition, child development and educa-

tion. It has also been found that, when using the au-

ditory modality in the education of hearing-impaired

children, onomatopoeia teaches vocalization patterns

and how to associate meaning with words, which fa-

cilitates verbalization (Nakamura, 2007). This sug-

gests that onomatopoeic words are important for the

hearing-impaired to learn. Other studies have found

that when deaf individuals understood spoken senten-

ces, not only the brain’s left hemisphere (as in na-

tive speakers with normal hearing) but also the right

brain was extensively activated (Neville, 1998), and

that deaf individuals used phonological representati-

ons in visually presented verbal memory tasks simi-

lar to people with normal hearing (Okada and Mat-

suda, 2015). However, when deaf subjects were as-

ked to judge the appropriateness of the use of sound

symbolic words depicting states, actions or emotions

to describe scenes in a video, the visual and audi-

tory association areas in the brain were not activated

(Arata and Matsuda, 2009). These ﬁndings support

the idea that it is difﬁcult for hearing-impaired in-

dividuals, who have difﬁculty accessing auditory in-

formation, to construct linguistic symbols using the

sound symbolism of onomatopoeic words, in spite

of their synesthetic and sound-symbolic characteris-

Namatame, M., Kusunoki, F. and Inagaki, S.

Basic Research on Multisensory Methods for Teaching Onomatopoeia to the Hearing-impaired.

DOI: 10.5220/0006665800220027

In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 22-27

ISBN: 978-989-758-291-2

tics (Hinton and Ohala, 2006). Thus, onomatopoeias

(also referred to as sound-symbolic words) are difﬁ-

cult for these individuals to learn. However, thinking

in another way, even without the ability to hold sound

symbols in memory in the absence of auditory infor-

mation, one can postulate that they may be able to

construct other linguistic symbols if a multisensory

environment is created to provide alternative sensory

information, such as vibrations or visual information.

In particular, hearing impairment has been linked to

a heightened sense of touch (Heidenreich and Le-

win, 2012). Previous research on information acces-

sibility and sensory substitution for the disabled has

shown that haptic information is effective in speech

training (Alves and Freire, 2015). Practical applica-

tions using vibrations to provide alerts are already

available, such as, vibrating alarm clocks, vibrating

devices worn in the hair (Honda, 2016), and applica-

tions that alert the user to sounds indicating danger

(Henriquez and Travieso, 2013). In addition, infor-

mation provided through vibrations enables hearing-

impaired individuals to recognize rhythms and enjoy

music (Kanebako and Namatame, 2016). This makes

the use of vibration promising as an alternative to au-

ditory information. Consequently, the author attemp-

ted to develop lesson which would enable people with

impaired hearing to learn sound symbols. A multisen-

sory lesson was designed to teach onomatopoeias mi-

micking animal sounds, which broadened learner ex-

perience beyond sound by simultaneously presenting

them with haptic and visual information in addition to

auditory information.

2 PURPOSE

The purpose of this study was to ﬁnd out what com-

ponent elements hearing-impaired learners appreciate

in a multisensory lesson designed to teach onomato-

poeias for animal sounds by broadening the learner

experience of sound and whether vibration as a form

of haptic information can play a complementary role

to auditory information in learning.

3 EXPERIMENTAL METHOD

In this report, we develop the teaching materials pro-

moting understanding for synesthesia about the kind

of the experience like insect (cicada) sound which the

hearing-impaired was not able to learn because acqui-

sition of the hearing information is difﬁcult.

We measure the learning effect by comparing the

cases when the vibration information is added or not

added. The evaluation of the way of receiving and the

teaching materials for the sense of touch (vibration)

will be obtained.

3.1 Design of Teaching Material

Because the body of the cicada is the housing, the best

way to learn the cicada’s song is the touching of a ci-

cada. However, it is difﬁcult to hear the original sound

because the cicada makes a different sound when the

cicada is touched. Therefore we designed the multi-

media teaching materials which appealed to hearing,

the sense of touch, sight to help the learning of the

sound of cicada.

We chose seven kinds of cicadas which are well-

known in Japan. The teaching materials contents are

constituted by the commentary sentence about the ha-

bits of the cicada, the onomatopoeia of the cicada’s

song, the images of cicada, the sound information of

the cicada’s song, the waveform of the cicada’s song,

vibration. In the main display, the commentary sen-

tence of the cicada and onomatopoeia and an image

and the button of the cicada’s song are shown.

Figure 1: Conﬁguration of the learning materials.

Basic Research on Multisensory Methods for Teaching Onomatopoeia to the Hearing-impaired

Table 1: Speciﬁcations of the cicadas’ sound.

cicada onomatopoeia time volume vibration

ms db m/s2

1 Jiri-jiri 45 95 2.8

2 Achi-achi 40 95 27.0

3 Kana-kana 46 93 9.2

4 chi. chi. chi. 41 91 3.7

5 Tsuku-tusku 42 95 3.3

6 min-min 46 95 3.0

7 Chiiii... 45 100 0.6

The sound waveform of the cicada’s song emerges

to a vice-display when a button is clicked, and the ci-

cada’s song is played back. The vibration links with

the cicada’s song by using a vibration speaker. The ci-

cada’s song is edited so that it becomes 93db(+-2db)

for approximately 40 seconds. In addition, the presen-

tation of the cicada in the learning using the teaching

materials is shown at random every time.

The presentation of the cicada is shown automa-

tically when the learner clicks the start button. The

learner can read the commentary sentence of the ci-

cada and onomatopoeia, and then click the button of

the cicada’s song. The learner get the cicada’s song

and the sound waveform with the vibration.

The constitution of the learning materials is shown

in ﬁgure.1, the sound condition of each cicada is

shown in table.1.

3.2 Experiment Procedure

The experiment was conducted from 27th Jun. until

6th Jul. in 2017. We had 26 participants with

hearing-impaired (Average age: 21.1 years old). The

participants were divided into two groups. Group A

was 17 participants using the vibration device, and

Group B was 9 participants not using the vibration

device. The experiment was 30 minutes in total per

person. Experiment procedure is following.

1: Informed consent and experiment explanation

2: 2back task (working memory task)

3: PreTest

4: Learning

5: Post Test

6: Questionnaire evaluation

3.3 Evaluation Method of the Lesson

This lesson used multiple media including text, ima-

ges, sounds, sound waveforms, onomatopoeia and vi-

brations to stimulate the senses of sight, hearing and

touch. The following 8 questions were prepared to

evaluate the lesson.

Q1: Good material

text, images, sounds, sound waveforms,

onomatopoeia, (vibrations)

Q2: Did you hear anything?

Q3: Did you distinguish those vibrations?

Q3’:(or Did you need vibration?)

Q4: The degree of easy intelligibility

Q5: The degree of useful

Q6: The degree of interesting

Q7: The degree of enjoyable

Q8: The degree of educative

4 RESULT

4.1 Learning Ability of Participants

As a baseline information, participants were assigned

the 2 Back-task, which you need to decide if the cur-

rent number is the same as the one presented 2 trials

ago, in order to assess the working-memory. After

a practice, formal experiment was conducted twice.

As the mean score of a percentage of two answers

were calculated, mean score of experimental group

was 53.2% and that of control group was 57.9%. In

order to assess the knowledge of cicada’s song, the

participants connected the 7 cicada’s names and 7 ci-

cada’s song (Onomatopoeia) as one by one, before

and after the learning. As a result, a percentage of

correct answers of pre-test was 3.9 for experimental

group and 3.8 for control group. That of post-test

was 6.8 for experimental group and 6.6 for control

group. That is, almost all the participants became

able to match the cicada’s name and song (Onoma-

topoeia) after the learning. Therefore, we regarded

the working-memory and leaning effect of the parti-

cipants as equal, and compared the subjective evalua-

tion about the lesson between the experimental group

and control group.

4.2 Evaluation About the Lesson

These ﬁgures (Figure 2Figure 3) show the percentage

that evaluation about good points of the lesson. The

vertical axis shows a percentage and horizontal axis

shows the alternatives. The alternatives are caption,

photo, onomatopoeia, sound, waveform and vibra-

tion. This question was multiple answers. The each

population were the 17 participants which gave vibra-

tion information, and the 9 participants which didn’t

give vibration information.

With or without vibration, we can read that the

onomatopoeia were evaluated as a factor to constitute

lesson. When there was not vibration, the sound was

estimated as an onomatopoeia at the same level, but

CSEDU 2018 - 10th International Conference on Computer Supported Education

Figure 2: Good element (Vib.).

Figure 3: Good element (no Vib.).

the sound was not evaluated too much when there was

vibration.

4.3 Subjective Evaluation About the

Lesson

The likert-scale was marked every one point and was

calculated for getting the average. The point of scale

was following.

+2point: strong-agree

+1point: agree

0point: not either

-1point: disagree

-2point: strong-disagree

The result shows the ﬁgure.4.

In order to check the differences between group

A (with vibration) and group B (without vibration),

T-test was analyzed. But there were not signiﬁcant

differences.

Figure 4: Comparison of the subjective evaluation.

4.4 Relation of the Sense

We made two questions about the sense of hearing

and touching. Q2Did you hear anything?, Q3Did you

distinguish those vibrations? The t-test was exami-

ned to provide signiﬁcant difference of their feeling

and subjective evaluation. The population was the 17

participants which gave vibration information. They

were separated into positive group and the negative

group. The positive-group was comprised of ”agree”

”strong-agree”, and the negative-group was compri-

sed of ”disagree”,”strong-disagree” and ”not either”.

4.4.1 Q2Did You Hear Anything?

The result that The 11 participants was ”can hear so-

mething” and the 6 participants was ”can’t hear any-

thing”. There was no signiﬁcant difference between

the subjective evaluation (IntelligibilityUseful, Inte-

restingEnjoyableEducative) and hearing-feeling. The

result shows that it was not different in the evaluation

of the lesson by the degree of the hearing.

4.4.2 Q3Did You Distinguish those Vibrations?

The result that The 11 participants was ”can distin-

guish” and the 6 participants was ”can’t distinguish”.

There were signiﬁcant differences between the

subjective evaluation about ”Interesting”t(15)=3.05,

p.05)and ”Educative”t(15)=2.49, p.05).

The table.2 shows that an evaluation is higher in

the person who was able to distinguish the vibration.

Basic Research on Multisensory Methods for Teaching Onomatopoeia to the Hearing-impaired

Table 2: Distinction of the vibration and evaluation of the

lesson.

distinguish no distinguish

AverageS.D. AverageS.D.

Intelligibility 1.09(0.70) 0.50(0.84)

Useful 1.27(0.47) 0.33(1.21)

Interesting 0.45(0.82) 0.17(0.75)

Enjoyable 1.27(0.65) 0.33(0.52)

Educative 1.00(0.77) -0.17(1.17)

5 DISCUSSION

The purpose of this study was to ﬁnd out whether a

multisensory lesson to teach sound symbolic onoma-

topoeias for animal sounds by broadening the lear-

ner experience of sound beyond auditory information

could facilitate learner comprehension. The subject of

the lesson was cicada songs. Analyses were based on

participants’ evaluations of the lesson and on tests to

determine if the lesson had an effect on participants’

learning .

Results showed no signiﬁcant difference in lear-

ning between two groups of participants (26 in total),

one that was provided with vibrations as part of the

lesson and one that was not. Post-lesson test results

showed improvement over the pre-lesson results with

almost all participants ultimately answering all the

questions correctly. However, to properly assess the

lesson’s effectiveness, knowledge retention needed to

be measured by conducting another test at some set

period of time after the lesson.

There were inconsistencies in how participants

were able to feel the vibrations in the lesson, which

suggested that the vibrations were not provided in a

way that everyone could understand them. Lesson

content consisted of an explanation of cicada ecology,

the onomatopoeia for the cicada’s song, a picture of

the cicada, the cicada’s song (auditory information),

its sound waveform and, for one of the groups, a vi-

bration.

Regardless of whether vibrations were included

in their group, most participants rated the onomato-

poeias for the songs as a good element in the les-

son. They thought they were useful for learning sound

symbolic words as they conveyed the sounds in word

form.

For the group in which vibrations were included

as an element in the lesson, fewer of the participants

selected the songs auditory information than the vi-

bration as a good element in the lesson, suggesting

that they may have tried to make more use of vibra-

tions when they were available. On the other hand,

when vibrations were not included, more of those par-

ticipants appreciated the inclusion of the songs audi-

tory information. This indicated that these hearing-

impaired individuals were using their residual hea-

ring.

No statistically signiﬁcant difference was found in

the subjective evaluations of the lesson between the

groups given and not given the vibrations. Nor was

there any signiﬁcant difference in evaluation results

due to the degree to which participants reported they

were able to hear sounds.

However, a signiﬁcant difference was shown due

to differences in how participants were able to feel

the vibrations. Those that were able to discriminate

among them reported they found the lesson fun and

that it could be used for teaching science. Given that

including vibrations in a lesson may make learning

sound symbolic words enjoyable, it may be promi-

sing to put them to use in science lessons for young

children with poor hearing.

6 CONCLUSION AND FUTURE

TOPICS

The elements of the multisensory lesson on onomato-

poeia that broadened learner experience of sound and

were most appreciated by the participants in this study

were the onomatopoeia, the sound waveforms for the

cicada songs, and either the song or its vibration, de-

pending on which group they were assigned to.

The study demonstrated that haptic information

like vibrations is a useful alternative to sound that

complements auditory information for people with

difﬁculty hearing. While results did not show that the

inclusion of vibrations in the lesson facilitated lear-

ner comprehension of sound symbolic onomatopoeia,

it did show vibration to be a promising element for

inclusion in educational materials for the hearing im-

paired.

In the future, it may be possible to develop more

effective lessons by using better vibration devices that

can vibrate in more distinctive ways. Better devices

are needed that can convey more nuances and higher

registers of sound, not vibrate the way an alarm does.

ACKNOWLEDGEMENTS

We thank to the students of the Tsukuba University of

Technology who cooperated on the experiment. We

thank J. Kanebako and M. Kitamura.

This work was supported by JSPS KAKENHI

Grant Number 15K12122.

CSEDU 2018 - 10th International Conference on Computer Supported Education

REFERENCES

Alves, R. and Freire, R. (2015). Correction of the funda-

mental voice frequency using tactile and visual feed-

back. In Proceedings of the 17th International ACM

SIGACCESS Conference on Computers & Accessibi-

lity. ACM.

Arata, M. and Matsuda, T. (2009). Semantic processing of

mimetic words in deaf individuals. an fmri study. In

26th O2-3,. Japanese Cognitive Science Society.

Heidenreich, M. and Lewin, R. (2012). Deafness linked to

heightened sense of touch. In Nature Neuroscience.

Springer Nature.

Henriquez, P. and Travieso, C. (2013). Review of automatic

fault diagnosis systems using audio and vibration sig-

nals. In TRANSACTIONS ON SYSTEMS, MAN, AND

CYBERNETICS: SYSTEMS. IEEE.

Hinton, L. and Ohala, J. (2006). Sound Symbolism. Cam-

bridge University Press, USA, 1st edition.

Honda, T. (2016). ontenna http://ontenna.jp/en/. Fujitsu,

Tokyo.

Imai, M. and Kita, S. (2014). The sound symbolism boots-

trapping hypothesis for language acquisition and lan-

guage evolution. In PUBMED. US National Library

of Medicine.

Kanebako, J. and Namatame, M. (2016). Effect of vibration

on listening sound for a person with hearing loss. In

Computers Helping People with Special Needs. Sprin-

ger Nature.

Miyazaki, A. and Tomimatsu, K. (2009). Onomato pla-

nets: physical computing of japanese onomatopoeia.

In Proceedings of the 3rd International Conference on

Tangible and Embedded Interaction. ACM.

Nakamura, K. (2007). Use of the auditory modality and

language acquisition. In The Journal Logopedics and

Phoniatrics (in Japanese). The Japan Society of Lo-

gopedics and Phoniatrics.

Neville, H. (1998). Cerebral organization for language in

deaf and hearing subjects. In Proc Natl Acad Sci USA.

National Center for Biotechnology Information.

Okada, R. and Matsuda, T. (2015). The deaf utilize pho-

nological representations in visually presented verbal

memory tasks. In Neuroscience Research. National

Center for Biotechnology Information.

Ono, M. (2007). Nihongo onomatope jiten (Japanese Mi-

metics Dictionary). Shogakkan, Tokyo, 1st edition.

Yamaguchi, N. (2003). Giongo gitaigo jiten (Onomato-

poeia Mimetic words Dictionary). Kodansha, Tokyo,

2nd edition.

Yusuf, M. and Watanabe, C. (2008). Onomatopeta!: De-

veloping a japanese onomatopoeia learning-support

system utilizing native speakers cooperation. In

Proceedings of the 2008 IEEE/WIC/ACM Internati-

onal Conference on Web Intelligence and Intelligent

Agent Technology. IEEE Computer Society Washing-

ton, DC, USA.

Basic Research on Multisensory Methods for Teaching Onomatopoeia to the Hearing-impaired