Automatic Objective Assessments of Japanese Reading Difficulty

with the Operation Records on Japanese Text Presentation System

Design and Implementation

Kyota Aoki

, Shinjiro Murayama

and Koji Harada

Graduate school of Engineering, Utsunomiya University, Yoto, Utsunomiya, Japan

Faculty of Education, Utsunomiya University, Mine, Utsunomiya, Japan

Keywords: Reading Difficulty, Text Presentation, Assessment.

Abstract: There are many pupils with reading difficulty in Japanese schools. The dyslexia is the disability about

reading and writing texts. We use Kanji, Hiragana, Katakana characters in Japanese sentences. Based on the

experiments, we show the differences between the reading activities with and without reading difficulties.

This enables to make objective assessments of reading difficulty. This paper proposes the objective

assessment method of reading difficulties and the design of automatic utterance detection with face images

that decrease the contributions of teachers.

1 INTRODUCTION

There are many pupils with reading difficulty in

Japanese elementary schools. There are many

difficulties.

The big and first one is reading Japanese

characters. Japanese characters are the construction

of hiragana (phonetic character), katakana (another

type of a phonetic character), kanji (Semantic

character) and other characters. In the period of

primary school, pupils learn 48 characters of

hiragana, 48 characters of katakana and 1008

characters of kanji. Almost all pupils learn hiragana

and katakana easily. However, the huge number of

kanji is difficult to learn for some pupils in normal

classes.

The next one is the difficulties about recognizing

the sentence structures. In Japanese sentences, there

is no spacing between words. For easing the

difficulties about reading kanji characters, we can

replace kanji characters with the hiragana characters.

We recognize the words constructing the Japanese

text in the help of kanji. There are a large number of

words starting from the character of kanji. We

recognize the chunk of characters that constructs a

word in the complex of hiragana, katakana and kanji.

Replacing kanji characters with hiragana

characters, we have the sequence of hiragana

characters only. In a long sequence of hiragana, it is

difficult to recognize the chunk of characters

constructing a word. In our experiments, the pupils

without reading difficulties like the longer

high-lighted regions than the pupils with reading

difficulties do. With the help of shorter high-lighted

regions, some pupils with reading difficulties read

more smoothly.

In an elementary school, pupils learn hiragana

and katakana at first. In the first stage in elementary

schools, the Japanese text-books have a space

between words for the ease of understanding the

structures of the sentences. However, normal

Japanese texts have no space between words.

Every pupil has those 2 difficulties at first. In the

long school life, they acquire the skill to conquer

those difficulties. Anyway, those 2 difficulties are

large barriers for reading and understanding

Japanese sentences.

Every infant has no knowledge about the

Japanese characters. Every pupil has a little

knowledge about the huge number of kanji

characters at first. Then, they learn hiragana,

katakana and kanji characters in a long

elementary-school life.

In Japanese elementary schools, reading

difficulty means 2 years delay of reading abilities. A

few of pupils with dyslexia learn in special support

education schools. However, there are many pupils

with reading difficulties in normal elementary

139

Aoki K., Murayama S. and Harada K..

Automatic Objective Assessments of Japanese Reading Difﬁculty with the Operation Records on Japanese Text Presentation System - Design and

Implementation.

DOI: 10.5220/0004839401390146

In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 139-146

ISBN: 978-989-758-021-5

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

Figure 1: DAISY visual presentation example.

schools. Of course, some pupils have difficulty about

remembering kanji characters. Most of the pupils

remember kanji characters gradually. However,

pupils with a learning disability tendency have

difficulty with reading Japanese sentences in the

case that they can remember the kanji characters. In

the case, they may have dyslexia.

There may be many causes of the difficulties on

reading Japanese texts. We do not discuss the causes.

We only pay attentions to the methods for easing

their difficulties. We call their difficulties “reading

difficulty” in this paper.

The research about teachers shows that the pupils

with ADSH tendency have difficulty about following

the characters sequentially and recognizing the

grammatical structures (Murayama, Aoki, 2009). Of

course, there are many types of reading difficulties.

There are many causes of the reading difficulties.

The resulting reading difficulties show the similar

symptoms. They are the difficulties about following

the characters sequentially, recognizing grammatical

structures and reading kanji characters.

We have developed a visual text presentation

system for persons with reading difficulty in

windows environments. The system records every

operation of the user. With the recorded operations,

we assess the difficulty of the user.

Figure 2: 1 Level annotation and 1 level mask with

background coloring.

The DAISY is one implementation of digital

talking book. Figure 1 shows an example of visual

presentation with DAISY. It includes annotation of

talking chunk of characters. It reads out the DAISY

contents. It has visual text presenting functions, also

(DAISY). The DAISY started for helping the people

with sight disability. The DAISY is helpful for pupils

with reading difficulty. However, the DAISY is a

digital talking book. The content must be carefully

prepared. Furthermore, there is a difficulty to make

DAISY contents from electrical texts. The advanced

teachers prepare the DAISY contents and use the

DAISY in their classes. However, the preparations

need a large amount of works. It is difficult to use

emerging materials in their classes for the large

preparation works. For example, it is difficult to use

the morning news in the class on the same day. The

interesting fresh materials can attract pupils’

attentions more. The DAISY does not offer such

functions. (DeMeglio, Hakkinen and Kawamura,

2002)

The Japanese text presentation system was

proposed and implemented for the pupils with

reading difficulties (Aoki, Murayama, 2012). The

system provides the multi-level annotation as shown

in figure 2. The system works well in a normal

elementary school (Murayama, Aoki, 2012). The

system makes the precise record of the operations.

With the operational record, we can assess the

reading difficulties on objective base.

This paper proposes the objective assessment

method of reading difficulties with the operational

record of the Japanese text presentation system, and

the system decreases the work of teachers who help

the pupils with reading difficulties.

First, this paper proposes the assessment method

using the system operational records. Then, we

discuss the problems for utilizing the assessment.

Next, we propose the automatic assessment method

for the pupils with reading difficulty. And last, we

conclude this work.

2 OBJECTIVE ASSESSMENTS

OF JAPANESE READING

DIFFICULTY

With the Japanese text presentation system, the

behavior of a user is recorded. From the recorded

behavior, we can assess the reading difficulties about

the user. This assessment is objective.

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

140

2.1 Operational Record of the Japanese

Text Presentation System

The Japanese text presentation system records all the

operations of a user. The Japanese text presentation

system moves the high-lighted part in a text with the

key-input of the user.

In using the Japanese text presentation system,

the user directs the move to the next high-lighted

part with a key-input. The record includes the key

operations with the precise time. With the record, we

can measure the time for reading the high-lighted

part. Table 1 shows the example of the operational

record. The 1st and the 2

columns have the date

and time. The 3

column is the operation. The 4th

column includes the high-lighted part. They are

Japanese. The 5th column does the elapsed time

from starting the system in 1/1000 seconds.

2.2 Assessment of Reading Difficulty

For assessing, we use the relation between the

reading time and the length of the high-lighted part.

There are the number of characters and the number

of phonemes for measuring the length of the text. In

our pre-experiments, it shows clear relations

between the reading time and the number of

characters. We use the number of characters for

measuring the length of a text. In Japanese texts,

there are kanji, hiragana and etc. As a result, there is

a change of phonemes in a character. However, the

number of character shows better relation to the

reading time.

The Japanese text materials differ in the target

age of the readers. For elder pupils, the materials

include more kanji characters. A single kanji

character represents a same word that is represented

using many hiragana characters. Their

pronunciations are same. The elder pupils read faster

than the younger pupils do. As a result, there are

constant relations between the number of characters

and the reading time of a material.

Without reading difficulties, there is a linear

relation between the reading time and the length of

the high-lighted part. However, in real reading, there

are many miss-operations and reading difficulties.

Figure 3 shows the example of the relation between

the reading time and the length of the high-lighted

part. There are points on a linear function and

out-lire points.

We decide the out-lire points in the reading time

per character of the high-lighted part. We use a

simple threshold for this process. We decide that the

reading time per character without reading

difficulties are between 0.1S and 0.3S. We plot the

pairs of the length and the reading time of the

high-lighted parts after filtering with the threshold in

figure 4.

In figure 4, the pairs of the length and the reading

time have the relation of linear function clearly. With

the reading difficulties, the pupil needs much more

reading time. As a result, the high-lighted parts

where the user has difficulties for reading are plotted

upper regions over the linear function.

We estimate the linear relation between the

reading time and the length of the high-lighted parts

from the data that are filtered with thresholds.

The data plotted in figure 4 have the relation in

(1).

Table 1: Example of the recorded operations.

Date Time Operation High-lighted part Elapsed time(mS)

2013/7/4 15:42:42 RIGHT1

テレビや新聞では、

12593

2013/7/4 15:42:45 RIGHT1

天気予報が毎日伝えられています。

15375

2013/7/4 15:42:46 RIGHT1

この天気予報は、

17015

2013/7/4 15:42:49 RIGHT1

たくさんの資料にもとづいて、

19265

2013/7/4 15:42:52 RIGHT1

専門家が天気の変化を予測したものです。

22984

2013/7/4 15:42:54 RIGHT1

わたしたちは、

24250

2013/7/4 15:42:55 RIGHT1

天気予報を見て、

25765

2013/7/4 15:42:57 RIGHT1

明日の予定を立てたり、

27640

2013/7/4 15:43:00 RIGHT1

持ち物を

30281

2013/7/4 15:43:01 RIGHT1

用意したり

31250

2013/7/4 15:43:02 RIGHT1

して

33047

2013/7/4 15:44:26 LEFT3

います。

116797

AutomaticObjectiveAssessmentsofJapaneseReadingDifficultywiththeOperationRecordsonJapaneseText

PresentationSystem-DesignandImplementation

141

Figure 3: Relation between reading time and the length.

Figure 4: Relation between reading time and the length

without out-lire data.

 177.1X  13.26

(1)

In (1), Y is the reading time in 1/1000 second. X is

the length of the high-lighted part in the number of

characters.

Figure 5: Assessment process of reading difficulty.

The plotted points over the normal linear

function direct the reading difficulties. The

corresponding part of the text shows the kinds of

reading difficulties. The data under the linear

function may be miss-operations. We decide that

Y  1000 is the range of normal reading activities.

The relation between the normal reading time

and the length of the high-lighted part differs in the

text and the school year. With large reading

experiments, we can decide the normal reading

activities. Figure 5 shows the process to assess the

reading difficulties.

3 PROBLES ABOUT READING

DIFFICULTY ASSESMENT

3.1 Teacher’s Contributions

The assessment process needs large amount of

teacher contributions. In reading with the Japanese

text presentation system, teachers monitor the

process of the readings. After that, teachers see the

operational records. This assessment results an

objective estimation of the reading difficulties of the

user. However, there is a little difference of the

teacher contributions between the assessment using

the Japanese text presentation system and the

classical assessment methods.

In the reading processes, a pupil may read the

part that is not high-lighted. A pupil may make

un-correct pronunciation. Those events make no

marks in the operational record. The observing

teachers guide the pupil for proper operations of the

Japanese text presentation system. The teachers also

record the un-correct pronunciations.

The Japanese text presentation system tries to

help every pupil with reading difficulties in a normal

class room. In Japanese elementary schools, there

are a few pupils with reading difficulties. The

teacher must make a class for the majority of normal

pupils. The teachers need the day by day

assessments of reading difficulties for evaluating

their teaching to ease the reading difficulties of a

pupil. With the present Japanese text presentation

system, teachers can assess the difficulties about

reading. However, the Japanese text presentation

system needs many works of teachers. For enabling

day by day assessments of reading difficulty, we

must decrease the teacher contributions for assessing

the reading difficulties.

500

1000

1500

2000

2500

3000

3500

4000

0 5 10 15 20 25

y=177.14x‐ 13.256

R²=0.901

500

1000

1500

2000

2500

3000

3500

4000

0 5 10 15 20 25

Start

Compute the reading

time per character

Select normal

reading activities

with thresholds

Decide reading

difficulties

Recorded

operations

Estimate linear

function of normal

reading activities

Start

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

142

3.2 Class Room Environments

There are many problems for utilizing the ITC

technology in Japanese elementary schools

(Murayama, Aoki, Morioka, 2009). The problems

are listed in Table 2. For solving the problems, the

proposed text presentation system treats only the

electronic text. In Japan, a law forces to prepare the

electronic readable texts of text books (Law). And,

there are many documents accessible through the

Internet. There is no paper document for an input in

the proposed system.

Many pupils may remember the full text of the

many times used materials as text books. Those

remembered materials cannot be used for evaluating

the reading performance of a pupil. The reading of

the materials cannot help to enforce the reading

abilities of the pupil.

In normal class rooms, many pupils use the

Japanese text presentation system simultaneously. In

the environments, many pupils read the text aloud.

With a simple embedded microphone in PC, we have

difficulties to separate each pupil’s voice.

Figure 6: Japanese text presentation system with automatic

assessments.

3.3 System Design

The proposed Japanese text presentation system has

only 2 functions. As the first row of Table 3, we

restrict the functions of the proposed system. The

proposed system has some performance

measurements function about pupil. This is

discussed in previous section. The teachers around

the pupil with reading difficulties need the objective

measurements of the performance of the reading

ability of the pupil. For this purpose, the proposed

system provides the operation logging function. The

operation logs describe the reading speed at each

meaningful chunk of characters.

The proposed Japanese text presentation system

enables to use one-time materials for measuring the

performance of a pupil. The real-time presentation

generation enables any new plain text materials at

any time.

This real-time presentation generation enables to

adapt the presentation for each pupil with different

reading difficulties. DAISY has no function about

adaptation for each pupil.

For adapting the variety of pupils’ ages and

disability grade, the presentation system has the

function to replace the un-studied kanji characters

with hiragana characters. The phonic hiragana

character is first studied character. There is a little

difficulty about reading hiragana.

The operations to the presentation system have

the information about the user. The proposed system

logs every operation with the time. This log

represents the fluency of the reader.

The new system adds the video of the face

images and the record of the voice to the operational

record. With the video of the face image, new

proposed system measure the mouth movement.

With the recorded voice, the teacher may check the

pronunciations afterward.

The proposed system has the features listed in

Table 3. The first and the second rows are new added

features. They decrease the contributions of a teacher

about using the Japanese text presentation system.

For wide use of the Japanese text presentation

system, the system does not need large-scale

contributions of teachers. The network problem is

important in Japanese schools. There is a large

limitation about the internet access. As a result, some

cloud based implementation cannot work. The

proposed system must work without the internet

access.

4 AUTOMATIC ASSESSMENT

SYSTEM IMPLEMENTATION

We decrease the teacher contributions with the

automatic evaluation of the reading process using a

microphone and a camera. The new system records

the face images and the voice. The new Japanese text

presentation system includes the original Japanese

text presentation system. The new system includes

the function to measure the mouth movement, the

function to measure the voice strength and the

function to make reading difficulty assessment as in

figure 6.

4.1 Measurements about Voice

It is easy to measure the strength of the voice of a

Basic Japanese

text

presentation

system

Mouth

movement

measurement

system

Voice strength

measurement

system

Reading

difficulty

estimation

system

AutomaticObjectiveAssessmentsofJapaneseReadingDifficultywiththeOperationRecordsonJapaneseText

PresentationSystem-DesignandImplementation

143

user in experimental environments. With the voice of

a single person, it is difficult to evaluate the precise

pronunciations. However, it is easy to evaluate the

strength of the voice. Figure 7 shows the wave

pattern recorded with the embedded microphone in

PC.

With the motions around the mouth position and

the strength of the voice, the system estimates the

reading activities of the user. This is much easier

than to estimate the precise pronunciations.

The voice strength measurement is easy to

implement. The system estimates the background

noise with the measurement of continuous recorded

sound. With filtered sound, we can easily estimate

the voice strength of the user in the experimental

environments. In figure 7, it is easy to detect the

region where the voice is recorded.

In normal class room, there are many other

sounds other than the voice of the user. In the

environments, it is not easy to separate the voice

among other voices and noises. We use the recorded

voice for checking the pronunciations by the

teachers.

Figure 7: An example of recorded voice.

4.2 Measurements of the Motion

of a Mouth

With the embedded video camera in a PC, the

system records the face images of the user. Figure 8

shows an example of the processed images. The

circles show the results of face and mouth detections.

The figure shows the face, the eyes and the mouth

clearly. The detections works well.

Figure 9 shows the measurements process of the

motion of a mouth. First, face detector finds the face

in an image. Then, eye detector finds the eyes. And

last, the mouth detector finds the mouth in the

restricted region with the detected face region and

eyes’ regions. As a result, mouth detector finds the

mouth well. Without restriction of the search region,

the mouth detector finds some number of candidates

of a mouth.

Table 2: Problems about ICT usability in a special aid

school in Japan.

There is a load concentration into the teacher, who is

good at ICT.

There are a few educational materials for the

DAISY.

They do not use the SAVE AS DAISY.

Using OCR for preparing educational materials for

pupils with a learning disability tendency, the

recognition errors make a large check and correct work.

There are large works for replacing difficult kanji

characters with hiragana.

Using classical ICT tools as the DAISY, we need to

prepare educational materials for each pupil who has a

different age and a different disability.

It is difficult to evaluate the performance.

Table 3: The plan for covering the problems.

New/Old Feature

New Automatic operation observations.

New

Automatic assessment of reading

difficulties.

Old

A collection of simple software is better

than complex multi-functional software.

Old Avoid the usage of OCR.

Old

An educational material presentation

system that does not need the special

material preparations.

New

An evaluation method/function for

evaluating the performance of a pupil.

The system measures the motions around the mouth

position. This detector is based on the face detector

included in OpenCV (OpenCV). The detector has

face direction compensation and face size

compensations with the basic face detector. And, the

eye detector estimates the eyes’ positions that are not

directly detected with the face detector. With the

system, we can have the face and 2 eyes position

continuously. With the positions both of a face and 2

eyes, we can easily estimate the mouth position

relatively from the eyes’ position. The mouth

detector is also provided with OpenCV. In

Figure 8: An example of processed images.

-2,00E-01

-1,00E-01

0,00E+00

1,00E-01

2,00E-01

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

144

compensated images, the mouth detector will work

well.

The face and mouth detector estimates the center

and the radius of the region. With the center and the

radius of the mouth’s region, we decide the region of

the mouth’s motion detection as the rectangle

circumscribing the circle describing the mouth

detection. The proposed system uses this rectangle as

the mouth’s region.

The change of a face position is compensated

with the position of the eyes. As a result, we can

easily measure the mouth motions. The system

measures the motion in the mouth region with the

difference between adjacent 3 frames using motion

template method. The method decides the mouth’s

region. The motion template method is also provided

with OpenCV.

With this mouth motion measurement, the system

estimates the reading activities of the user very

precisely. With only the voice caught by a

microphone, in normal classroom environment, the

strength only of a voice is difficult to measure. The

observation of mouth form is also difficult in normal

classroom environment. However, the mouth

movement only is easy enough to measure. The

system measures the activity around a mouth using

the motion template method.

Figure 10 shows the measured mouth’s motions

and utterances. The dotted line shows the utterance.

The solid line shows the measured mouth’s motions.

The measured mouth’s motion is the number of

moving pixels in the mouse region. With the

measured mouth’s motions, we can easily estimate

the utterance of a user.

Figure 9: Mouth movement measurement.

The mouth motion measurement is robust about the

changes of the environments. In class rooms, there

must be proper lightings. Using the Japanese text

presentation system, the user faces the display. There

are no objects between the face and the display. The

camera is on the top rim of the display. As a result,

there is a little interception about recording the face

images of the user.

Figure 10: Measured mouth movements.

4.3 Estimation of Reading Activity

The voice of the user is easy to record using the

embedded microphone in PC. It is easy to handle the

recorded voice. However, in normal class rooms,

there are many noises. In the environments, it is

difficult to distinguish the voice of the user among

many noises.

The measurement of the mouth activity is more

difficult than the voice measurements, and it is time

consuming. However, face image acquisition is easy

and robust in the normal class room environments.

Our proposed system estimates the reading

activity of the user with the video measurements of

the mouth activity. When the mouth’s motion is

detected, the proposed system decides that the

reading activity exists at the time. With this

processing, the proposed system results the

estimated reading activities shown in figure 11. In

the figure, the dotted line shows the reading activity.

The solid line does the estimated reading activity.

The error rate is 0.094 in our experiments. In a

reading period, the error rate is 0.064. In a pausing

period, the error rate is 0.14. With these error rate,

we can check and control the operations of a user.

This implementation does not work fast enough

to process in real time environments. Our

implementation can process about 10 frames in 1

Face detection Eyes detection

Mouth detection

Mouth motion

measurement

Mouth region

estimation

Haar detector

Motion template

method

Haar detector

AutomaticObjectiveAssessmentsofJapaneseReadingDifficultywiththeOperationRecordsonJapaneseText

PresentationSystem-DesignandImplementation

145

second. The normal video is 33 frames in 1 second.

This processing speed enables to detect the utterance

for teacher’s checks of the automatic assessments.

However, we need much more performance for

assisting the user directly.

Figure 11: Estimated reading activities.

5 CONCLUSIONS

The proposed Japanese text presentation system

reports the precise reading activities of the user to

the teacher. The report includes not only the key

operations, but also the timing of utterance with

video analysis. The utterance timing is a direct

description of the user’s reading activities.

The proposed Japanese text presentation system

with automatic assessment enables to be used

simultaneously in a class room. In normal class room,

a teacher controls many pupils, including ones with

reading difficulties.

The new proposed system decreases the

contributions of a teacher for using the Japanese text

presentation system in a class. Teachers do not need

to check all record of the user’s reading activities.

The system detects the points where the reading

difficulty is. This enables easier use of the Japanese

text presentation system in normal class rooms.

The much more precise record of the user’s

activities helps to evaluate the precise assessment of

the reading difficulties with less teacher’s work.

The next step of this research is a wide

distribution of the software. However, this needs not

only the software distribution, but also the

distribution of the method to use the software

effectively.

ACKNOWLEDGEMENTS

We thanks Mr. Sho Kubota for his implementation of

the mouth’s motion measurement. This work is

supported with JSPS25330405.

REFERENCES

DeMeglio, M., Hakkinen, M., Kawamura,H., 2002.

Accessible Interface Design: Adaptive Multimedia

Information System (AMIS). Computers Helping

People with Special Needs, Lecture Notes in Computer

Science. Springer.

Law.http://www.bunka.go.jp/chosakuken/pdf/tokuteitosyo

_fukyu_gaiyo.pdf.

Murayama, S., Aoki, K., 2010. Real Time Image

Presentation System for Persons with a

Learning-Disabled Tendency. IEICE-ET, vol. 109, no.

387, ET2009-96, pp. 25-29, 2010. IEICE.

DAISY. http://www.daisy.org/

Murayama, S., Aoki, K., Morioka, N., 2009. Image

processing to make teaching aids for learning disability

persons. IEICE-108, IEICE-WIT-488, IEICE.

Aoki, K., Murayama, S., 2012. Japanese Text Presentation

System for Persons with Reading Difficulty -Design

and Implementation-. CSEDU2012, vol.1, pp. 123-128,

Porto.

Murayama, S., Aoki, K., 2012. Japanese Text Presentation

System for pupils with Reading Difficulties, ICCHP

2012, Lecture Notes in Computer Science, vol. 7382,

Computers Helping People with Special Needs,

pp.507-514, Linz, Austria.

OpenCV. http://opencv.org/.

CSEDU2014-6thInternationalConferenceonComputerSupportedEducation

146