Digital Assisted Communication

Paula Escudeiro, Nuno Escudeiro, Marcelo Norberto, Jorge Lopes, Fernando Soares

2017

Abstract

The communication with the deaf community can prove to be very challenging without the use of sign language. There is a considerable difference between sign and written language as they differ in both syntax and semantics. The work described in this paper addresses the development of a bidirectional translator between several sign languages and their respective text, as well as the evaluation methods and results of those tools. A multiplayer game is using the translator is also described on this paper. The translator from sign language to text employs two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. This translator is being adapted to allow the communication with the blind as well. The Quantitative Evaluation Framework (QEF) and the ten-fold cross-validation methods were used to evaluate the project and show promising results. Also, the product goes through a validation process by sign language experts and deaf users who provide their feedback answering a questionnaire. The translator exhibits a precision higher than 90% and the projects overall quality rates are close to 90% based on the QEF.

References

  1. Morgado, M., Martins, M., 2009. Língua Gestual Portuguesa. Comunicar através de gestos que falam é o nosso desafio, neste 25. º número da Diversidades, rumo à descoberta de circuns-tâncias propiciadoras nas quais todos se tornem protagonistas e interlocutores de um diálogo universal., 7.
  2. Morrissey, S., Way, A., 2005. An example-based approach to translating sign language.
  3. Gameiro, J., Cardoso, T., Rybarczyk, Y., 2014. KinectSign, Teaching sign language to “listeners” through a game. Procedia Technology, 17, 384-391.
  4. Biau, G., 2012. Analysis of a random forests model. Journal of Machine Learning Research, 13(Apr), 1063-1095.
  5. Cooper, H., Pugeault, N., & Bowden, R., 2011. Reading the signs: A video based sign dictionary. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on (pp. 914-919). IEEE.
  6. Steinwart, I., & Christmann, A., 2008. Support vector machines. Springer Science & Business Media.
  7. Brashear, H., Zafrulla, Z., Starner, T., Hamilton, H., Presti, P., & Lee, S., 2010. CopyCat: A Corpus for Verifying American Sign Language During Game Play by Deaf Children. In 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies.
  8. ProDeaf, 2016. Solutions. [Online] Available at: http://www.prodeaf.net/en-us/Solucoes [Accessed December 2016].
  9. Showleap, 2015. Showleap blog. [Online] Available at: http://blog.showleap.com/2015/12/2016-nuestro-ano/ [Accessed December 2016]. 79.
  10. Showleap, 2016. Showleap. [Online] Available at: http://www.showleap.com/ [December July 2016].
  11. Motionsavvy, 2016. Motionsavvy Uni. [Online] Available at: http://www.motionsavvy.com/ [Accessed December 2016].
  12. University of Washington, 2016. UW undergraduate team wins $10,000 Lemelson-MIT Student Prize for gloves that translate sign language. [Online] Available at: http://www.washington.edu/news/2016/04/12/uwundergraduate-team-wins-10000-lemelson-mitstudent-prize-for-gloves-that-translate-sign-language/ [Accessed December 2016].
  13. Gloves, 5DT., 2011. Fifth dimension technologies. [Online] Available at: http://www.5dt.com/?page_id=34.
  14. Zhang, H., Berg, A. C., Maire, M., & Malik, J., 2006. SVMKNN: Discriminative nearest neighbor classification for visual category recognition. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on (Vol. 2, pp. 2126-2136).
  15. Gee, J. P., 2003. What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20-20.
  16. Escudeiro, P., Escudeiro, N., Reis, R., Barbosa, M., Bidarra, J., Baltasar, A. B., ... & Norberto, M. (2014). Virtual sign game learning sign language. In A. Zaharim, & K. Sopian (Eds.), Computers and Technology in Modern Education, ser. Proceedings of the 5th International Conference on Education and Educational technologies, Malaysia.
Download


Paper Citation


in Harvard Style

Escudeiro P., Escudeiro N., Norberto M., Lopes J. and Soares F. (2017). Digital Assisted Communication . In Proceedings of the 13th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST, ISBN 978-989-758-246-2, pages 395-402. DOI: 10.5220/0006377903950402


in Bibtex Style

@conference{webist17,
author={Paula Escudeiro and Nuno Escudeiro and Marcelo Norberto and Jorge Lopes and Fernando Soares},
title={Digital Assisted Communication},
booktitle={Proceedings of the 13th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,},
year={2017},
pages={395-402},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006377903950402},
isbn={978-989-758-246-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 13th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,
TI - Digital Assisted Communication
SN - 978-989-758-246-2
AU - Escudeiro P.
AU - Escudeiro N.
AU - Norberto M.
AU - Lopes J.
AU - Soares F.
PY - 2017
SP - 395
EP - 402
DO - 10.5220/0006377903950402