Precise Estimation of Reading Activities with Face Image and Read Aloud Voice

Kyota Aoki, Shuichi Tashiro, Shu Aoki

2017

Abstract

In Japanese public primary schools, every pupil may use an ICT device individually and simultaneously. In normal primary school, a few teachers must teach all pupils in a class. It is difficult to help all pupils to use an ICT device. For using ICT devices individually in a normal class, the ICT device help its’ user by itself. To help the user, the ICT device must understand the state of the user. To help a teacher, it must precisely understand the users' reading activities. Reading ability is the base of all subjects. It is important that pupils acqure reading ability. This paper proposes a method to recognize the precise reading activity of a user with read aloud voices and facial images, shows its implementation, and experimental results. With the cooperative analysis of a read aloud voice from a microphone and a movement of a mouth from a camera, our implementation enables to estimate the action of read aloud much more precisely. The timing of a read aloud action is estimated in phrase by phrase manner. In-vitro experiments confirm the performance of our implementation.

References

  1. Aoki, K., Murayama, S. and Harada, K. (2014). Automatic Objective Assessments of Japanese Reading Difficulty with the Operation Records on Japanese Text Presentation System. CSEDU2014, vol. 2, pp.139-146, Barcelona, Spain.
  2. Aoki, K. and Murayama, S. (2012). Japanese Text Presentation System for Persons With Reading Difficulty -Design and Implementation-. CSEDU2012, vol.1, pp. 123-128, Porto, Portugal.
  3. Aoki, K., Murayama S., Aoki, S. and Tashiro, S. (2016). Recognition of Reading Activities and Reading Profile of User on Japanese Text Presentation System, Computer Supported Education, 7th International Conference, CSEDU 2015, Lisbon, Portugal, May 23- 25, 2015, Revised Selected Papers, pp. 57-80. Springer.
  4. Aoki, K., Tashiro, S. and Aoki, S. (2016). PRECISE UNDERSTANDIG OF READING ACTIVITIES - Sight, Aural, and Page turning-”, 8th International Conference on Computer Supported Education, Rome, Italy, April.
  5. Julius, (2016). https://github.com/julius-speech/julius.
  6. Levenshtein A. (1966). Binary Codes Capable of Correcting Deletions, Insertions and Reversals, Soviet Physics Doklady, vol. 10, no. 8, pp. 707-710.
  7. Mecab (2016). http://mecab.googlecode.com/svn/trunk/mecab/doc/ind ex.html?sess=3f6a4f9896295ef2480fa2482de521f6.
  8. OpenCV (2016). http://opencv.org/.
  9. Pyglet (2016). http://pyget.com/about.html.
  10. Python (2016). https://www.python.org/.
Download


Paper Citation


in Harvard Style

Aoki K., Tashiro S. and Aoki S. (2017). Precise Estimation of Reading Activities with Face Image and Read Aloud Voice . In Proceedings of the 9th International Conference on Computer Supported Education - Volume 1: CSEDU, ISBN 978-989-758-239-4, pages 315-322. DOI: 10.5220/0006315603150322


in Bibtex Style

@conference{csedu17,
author={Kyota Aoki and Shuichi Tashiro and Shu Aoki},
title={Precise Estimation of Reading Activities with Face Image and Read Aloud Voice},
booktitle={Proceedings of the 9th International Conference on Computer Supported Education - Volume 1: CSEDU,},
year={2017},
pages={315-322},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006315603150322},
isbn={978-989-758-239-4},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 9th International Conference on Computer Supported Education - Volume 1: CSEDU,
TI - Precise Estimation of Reading Activities with Face Image and Read Aloud Voice
SN - 978-989-758-239-4
AU - Aoki K.
AU - Tashiro S.
AU - Aoki S.
PY - 2017
SP - 315
EP - 322
DO - 10.5220/0006315603150322