Communication Support for the Deaf and Hard of Hearing using

Head Mounted Display: Toward Its Use in Museums

Nobuko Kato

and Yuto Shinozaki

Faculty of Industrial Technology, Tsukuba University of Technology, Tsukuba, Ibaraki, Japan

Keywords: Deaf and Hard of Hearing Person, Communication Support, Closed Captions, Accessibility.

Abstract: The purpose of this study was to clarify the issues of using Optical See-Through Head Mounted Displays

(HMD) to provide communication support for the disabled in museums. The study also examined how to

present closed captions in the HMD that would be suitable for the deaf and hard of hearing. The presentation

of closed captions for the deaf and hard of hearing person adopts different methods, depending on the situation,

such as presenting a few lines (akin to closed captions used in television programs and movies) or presenting

several dozen lines in an entire screen, as seen in university classes and lectures. There are many items that

need to be examined in order to use HMD in museums, including the ideal HMD presentation method for the

deaf and hard of hearing person and the comprehension levels and safety when closed captions are presented.

This paper detailed the results of a simulation experiment of presenting closed captions using HMD for the

deaf and hard of hearing.

1 INTRODUCTION

In recent years, museums have played an important

role as a place for science education in local

community (Gilbert, 2012). For the deaf and hard of

hearing to have an efficiency experience in a

museum, some kind of support is needed to

supplement auditory information, such as creating

learning content (Constantinou, 2016) and preparing

guided tours for the deaf and hard of hearing

(Namatame, 2019).

Advances in augmented reality (AR) technology

that adds information to the real world (Azuma, 2001,

Kruijff, 2010), particularly with regard to advances in

resolution improvement of Head Mounted Displays

(HMD) and speech recognition technologies, have led

to research on the presentation of closed captions

using HMD (Kurahashi, 2018). However, research on

the cognitive load and comprehension levels of users

when various information is presented to the deaf and

hard of hearing, or users’ safety when they move

about while wearing one, has not yet been sufficiently

conducted.

There is a guideline that limits the number of

Japanese characters to 240 per minute to be displayed

when audience members read text while watching

http://www.tsukuba-tech.ac.jp/

films. Meanwhile, an average of 300-400 characters

are spoken in a minute when talking in Japanese at a

regular speed. Therefore, when transposing the

content spoken to written words, as they are, it will

require displaying many more characters, thereby

surpassing the character count limit used in films. For

this reason, when presenting the same amount of

information provided through auditory means in real-

time into characters or text for the deaf and hard of

hearing at museums and similar locations, it is

necessary to examine whether or not the deaf or hard

of hearing person can simultaneously view the

exhibits while processing the textual data. It is also

necessary to determine whether or not the deaf or hard

of hearing person can understand the content from the

presented closed caption while viewing the exhibits.

Because there is a trade-off between real-time

performance and the amount of information, it is

important to find an appropriate balance (Figure 1).

This research presumed a scene for providing

information support at a museum by using an optical

see-through HMD for the disabled in the museum.

The aim was to clarify issues related to HMD use and

to examine the presentation method of closed

captions in HMDs that are most suitable for the deaf

and hard of hearing.

Kato, N. and Shinozaki, Y.

Communication Support for the Deaf and Hard of Hearing using Head Mounted Display: Toward Its Use in Museums.

DOI: 10.5220/0009394005950600

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 1, pages 595-600

ISBN: 978-989-758-417-6

595

Figure 1: Issue of communication support using closed caption in Japanese.

2 EXPERIMENTS USING HMD

2.1 Examination of HMD Suitable for

Presenting Closed Captions in

Japanese

When closed captions are presented to an HMD, the

line count as well as the presentation position need to

be determined. A comparative review was conducted

between Sony's optical see-through eyeglass device,

SmartEyeglass SED-E1, Epson’s SmartGlass

MOVERIO BT -200, and BT -300 (Table 1). As a

result of a display trial, the BT-300 was selected as

the device to be used for the experiment since a high-

resolution is required to display several lines of

Japanese text, including Kanji characters.

2.2 Line Count and Presentation

Position

Table 1: Comparison of optical see-through HMDs.

Model

Name

Resolution Number of

colors

Angle of

view

(opposite

angle)

SED-E1 419 x 138 256(Green) 20 degrees

BT-200 960 x 504 Approx.

1.677M

Approx. 23

degrees

BT-300 1280 x 720 Approx.

1.677M

23 degrees

Table 2: The rank sum of the comparison experiment.

Positions and line count Rank sum

Upper 1 line 25

Lower 1 line 21

Upper 3 lines 17

Lower 3 lines 17

2.2.1 Comparison Experiment

In order to compare the presentation position and line

count of closed captions, a total of four types of

closed captions were prepared, namely closed

captions in the 1) upper or 2) lower positions of the

screen, with the line count set as 3) one or 4) three.

The four types were presented on HMDs in a random

order, with the eight experiment participants ranking

them in order of 1st to 4th in terms of ease of viewing.

2.2.2 Result of Experiment

The rank sum of the eight participants’ experiences

with the four types of closed captions (i.e., upper 1

line, upper 3 lines, lower 1 line, and lower 3 lines, in

a random order) is presented in Table 2. The results

showed that the rank sum of the upper 3 and lower 3

lines (rank: 17) were high, with the upper 1 line (rank:

25) and lower 1 line (rank: 21) being low.

The free writing sections in the post-experiment

survey indicated:

 3 lines were easy to see.

 Closed captions with many, rather than a few,

characters can be read with more ease.

CSEDU 2020 - 12th International Conference on Computer Supported Education

596

 Three lines were easier to navigate as they

allowed for participants to read the lines back.

Between closed captions with one line and three

lines, those with three lines tended to gain a high

evaluation; however, there was no difference between

the upper three lines and the lower three lines. The

difference in the preference between the upper and

lower positions is believed to be largely attributed to

the individual differences in how the closed captions

were seen, dependent on such factors as where the

HMD nose pads were positioned.

2.3 Closed Caption Presentation

Experiment using HMD

2.3.1 Method of Closed Caption

Presentation Experiment

As a simulation of a viewing experience at a museum,

an experiment was conducted in which participants

viewed, in order, four images simulating an

exhibition in a room while reading the closed captions

presented to the HMD. The following two types of

closed captions of explanatory notes of the exhibition

were presented:

 Closed captions that presented the original text

as they were.

 Closed captions that presented the simplified

text.

Here, the simplified text refers to a structured and

simplified version of the original text. The number of

characters was set at around 80% of the original text,

and structuring included aspects such as adding a

heading that indicated the content being explained.

The experiment participants were eight deaf or

hard of hearing students in their 20s. The experiment

was performed twice, with each participant

experiencing the original and simplified texts in

random order. One experiment session took about 3

minutes. A comprehension test was given before and

after each experiment, and a survey was conducted

after the experiment. The closed captions presented

on HMDs during the experiment displayed three

lines.

After conducting a test to enable the participants

to become acquainted with the closed caption

presentation on the HMDs, each participant was

asked to select the presentation position they

preferred (i.e. upper or lower). Their chosen selection

was used in the individual experiments.

2.3.2 Comprehension Test Results Related

to the Closed Captions Presentation

Participants’ comprehension test results from the

aforementioned experiment are presented in Figure 2.

Each comprehension test was scored out of 20.

The difference in comprehension test scores

before and after the test was 4.5 points on average for

the original text, and 5.6 points on average for the

simplified text. There was no significant difference

between the mean scores of the original and

simplified texts.

2.3.3 Survey Results

The results of the survey conducted after the

experiment are presented in Figures 3 and 4. The

mean evaluation score for the question “Do you think

explanations using closed captions is useful?”

(responses were given on a 7-point scale of 1:

Strongly disagree, and 7: Strongly agree) was 5.4 for

the original text and 5.8 for the simplified text

presentations (p<0.05).

The mean evaluation score for the question

“Would you like to use such closed captions again?”

(responses were given on a 7-point scale of 1:

Strongly disagree, and 7: Strongly agree) was 4.8 for

the original text and 5.1 for the simplified text

(p<0.05).

2.3.4 Free-writing Section

Surveys, which were conducted after the participants

experienced closed captions presentations using

HMD, included the following comments:

Descriptions related to reading the closed captions

while viewing the exhibits:

 Because I was focused on the closed captions,

my attention did not go much toward the

images.

 Although I wanted to see both the images and

the closed captions, I was only able to see one,

making it difficult to deepen my understanding.

 I personally wanted time to view the object

(image).

 The closed caption kept being presented; there

was no break.

 It was difficult to read the presented closed

captions while moving.

 I thought that it bugged me a little that I wasn’t

able to rewind the closed caption when I wanted

to see it again.

Communication Support for the Deaf and Hard of Hearing using Head Mounted Display: Toward Its Use in Museums

597

Participant ID

(a) Original text

Participant ID

(b) Simplified text

Figure 2: The comprehension test result before and after the experiment.

Descriptions related to viewing the simplified

text:

 It was easy to understand because the features

(of the exhibit) were separated by brackets at

the beginning of the sentence.

 It was easy to understand as there were sections

that were classified with headings like

[history].

The results related to participants’ preference for

using a Smartphone or HMD when moving, are

presented in Figure 5.

3 DISUCUSSION

3.1 The Line Count and Presentation

Position of the Closed Captions

Concerning the line count of closed captions, three

lines tended to receive a high evaluation, even in the

comments presented in the free-writing section of the

survey. Previous studies related to television screens

also showed that three lines of closed captions

received a high evaluation in Japanese (Monma,

2000). When asked about the best subtitle change-

over method, participants in such studies also

preferred three lines of closed captions completely

switching over, as opposed to captions scrolling

(Monma, 2000). Similar results were found with

HMD in this study.

CSEDU 2020 - 12th International Conference on Computer Supported Education

598

Participant ID

Figure 3: Survey (1) result: “Do you think explanations using such closed captions is useful?”.

Participant ID

Figure 4: Survey (2) result: “Would you like to use such closed captions again?”.

3.2 Relationship between Closed

Captions Presentations and

Comprehension Levels

As a result of the experiment that used both original

and simplified texts for closed captions presentations,

the free-writing section of the survey indicated that

participants had difficulties in viewing both the

closed captions and the images simulating a museum

exhibit. Although providing closed captions near the

exhibit is believed to reduce the gaze movement, the

results indicated that this practice does not eliminate

the difficulties.

A comparison of the original and simplified texts

answered the questions "Do you think explanations

using such closed captions are useful?" and "Would

you like to use such closed captions again?". These

answers tended to more highly evaluate the reading

of the simplified texts. The results could possibly be

attributed to how the time necessary to view the

exhibits was secured by reducing the amount of time

necessary to read the closed captions.

Conversely, no significant differences were found

in the comprehension tests. Thus, it can be concluded

that a certain level of content comprehension can be

obtained even if texts are properly simplified. The

free-writing section also indicated the efficacy of

structuring texts using such means as adding

headings.

Survey answers related to the question of whether

Smartphones or HMDs are easier to use when moving

indicated differing opinions. Three participants chose

Smartphones while five preferred HMDs. In

particular, those the deaf and hard of hearing who

tended to rely more heavily on closed captions than

audio during daily television watching tended to

choose Smartphones.

Communication Support for the Deaf and Hard of Hearing using Head Mounted Display: Toward Its Use in Museums

599

Figure 5: Survey result regarding the ease of viewing while moving.

4 CONCLUSIONS

In this study, a basic examination of a method of

presenting explanatory content as closed captions on

HMDs when viewing museum exhibits was

conducted. It was found that three lines of closed

captions tended to receive a higher evaluation than

one line in terms of presenting text to the deaf and

hard of hearing.

Furthermore, in an experiment that simulated

museum exhibits viewing, it was found that

participants had difficulties with viewing both the

target object and the closed captions. In addition, as

a result of the comparison that used both original and

simplified texts for closed captions presentations for

the deaf and hard of hearing, it was found that there

was no significant difference in the level of

comprehension, and that the simplified text had

higher evaluation. In order to see the exhibition and

the closed captions while moving in the room, it is

important to make a suitable time interval when the

closed captions are not displayed.

ACKNOWLEDGEMENTS

This work was supported by JSPS KAKENHI Grant

Number JP 18H03660.

REFERENCES

Gilbert, J.K., Stocklmayer, S.M., 2012. Communication

and Engagement with Science and Technology: Issues

and Dilemmas A Reader in Science Communication,

Routledge, UK.

Constantinou V., Loizides F., Ioannou A. (2016) A

Personal Tour of Cultural Heritage for Deaf Museum

Visitors. In: Ioannides M. et al. (eds) Digital Heritage.

Progress in Cultural Heritage: Documentation,

Preservation, and Protection. EuroMed 2016. Lecture

Notes in Computer Science, vol 10059. Springer,

Cham, 214-221.

Namatame M., et.al, 2019, Can Exhibit-Explanations in

Sign Language Contribute to the Accessibility of

Aquariums? , Proceedings of HCI International 2019

Posters, 289-294.

R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, B.

MacIntyre, 2001, Recent advances in augmented

reality, IEEE Comput. Graphics Appl., 21 (2001), pp.

34-47

Kruijff, E., Swan, J. E., S. Feiner, 2010, Perceptual issues

in augmented reality revisited, Proceedings of IEEE

International Symposium on Mixed Augmented

Reality, 3-12.

Kurahashi,T., Sakuma, R., Zempo, K., et.al., 2018,

Retrospective Speech Balloons on Speech-Visible AR

via Head-Mounted Display, IEEE International

Symposium on Mixed and Augmented Reality, 423-

424.

Monma,T., et al. , 2000, Subjective Evaluation on Closed

Caption Presentation Methods of News for the Deaf

and Hard of Hearing, The Journal of the Institute of

Image Information and Television Engineers,

54,9,1288-1297.

CSEDU 2020 - 12th International Conference on Computer Supported Education

600