Real Time Bidirectional Translator of Portuguese Sign Language

Paula Escudeiro, Nuno Escudeiro, Rosa Reis, Pedro Rodrigues, Jorge Lopes, Marcelo Norberto, Ana Bela Baltasar, Maciel Barbosa, José Bidarra

2015

Abstract

The communication with deaf by means of written text is not as efficient as it might seem. In fact, there is a very deep gap between sign language and spoken/written language. The deployment of tools to assist the daily communication between deaf people and the rest may be a significant contribution to the social inclusion of the deaf community. The work described in this paper addresses the development of a bidirectional translator between Portuguese Sign Language and Portuguese text and a serious game to promote the learning of the Portuguese Sign Language. The translator from sign language to text employs two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. The hands configurations are classified using Support Vector Machines. The classification of the movement and orientation of the hands is achieved through the use of Dynamic Time Warping algorithm. The translator exhibits a precision higher than 90%. In the other direction, the translation of Portuguese text to Portuguese Sign Language is supported by a 3D avatar which interprets the entered text and performs the corresponding animations. As a complement, we also present a serious game directed to assist in the difficult task of learning the Portuguese Sign Language.

References

  1. Larsson, Stig. "A Review of Claes G. Olsson: Omsorg och kontroll-en handikapphistorisk studie 1750-1930 (Care and control-an analysis of the history of disability) Umeå 2010." Vulnerable Groups & Inclusion 2 (2011).
  2. Associação de Surdos do Porto - Algumas definições úteis sobre a Surdez, http://www.asurdosporto .org.pt/artigo.asp?idartigo=77, January 2015.
  3. Morgado, Marta, and Mariana Martins. "Língua Gestual Portuguesa." Comunicar através de gestos que falam é o nosso desafio, neste 25. º número da Diversidades, rumo à descoberta de circunstâncias propiciadoras nas quais todos se tornem protagonistas e interlocutores de um diálogo universal.: 7.
  4. Morrissey, Sara, and Andy Way. "An example-based approach to translating sign language." (2005).
  5. Gameiro, João, Tiago Cardoso, and Yves Rybarczyk. "Kinect-Sign: Teaching Sign Language to “Listeners” through a Game." Innovative and Creative Developments in Multimodal Interaction Systems. Springer Berlin Heidelberg, 2014. 141-159.
  6. Deusdado, Leonel Domingues. Ensino da língua gestual assistido por personagens 3D virtuais. Diss. Universidade do Minho, 2002.
  7. Bragatto, T. A. C., G. I. S. Ruas, and M. V. Lamar. "Realtime video based finger spelling recognition system using low computational complexity Artificial Neural Networks." Telecommunications Symposium, 2006 International. IEEE, 2006.
  8. Pugeault, Nicolas, and Richard Bowden. "Spelling it out: Real-time asl fingerspelling recognition." Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011.
  9. Rezende, Lorena A., and Dalton M. Tavares. "OpenNI e Suas Aplicações." X Encontro Anual de Computação (2013).
  10. Breiman, Leo. "Random forests." Machine learning 45.1 (2001): 5-32.
  11. Cooper, Helen, Nicolas Pugeault, and Richard Bowden. "Reading the signs: A video based sign dictionary." Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on. IEEE, 2011.
  12. Vladutu, Liviu. "Non-rigid shape recognition for sign language understanding." WSEAS TRANSACTIONS on SYSTEMS 8.12 (2009): 1263-1272.
  13. Klema, Virginia, and Alan J. Laub. "The singular value decomposition: Its computation and some applications." Automatic Control, IEEE Transactions on 25.2 (1980): 164-176.
  14. Klir, George, and Bo Yuan. Fuzzy sets and fuzzy logic. Vol. 4. New Jersey: Prentice Hall, 1995.
  15. Steinwart, Ingo, and Andreas Christmann. Support vector machines. Springer, 2008.
  16. McGuire, R. Martin, et al. "Towards a one-way American sign language translator." Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE International Conference on. IEEE, 2004.
  17. Brashear, H., et al. "CopyCat: A Corpus for Verifying American Sign Language During Game Play by Deaf Children." 4th Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies. 2010.
  18. 5DT Data Glove Ultra - User's Manual, http://www.5dt.com/downloads/dataglove/ultra/5DT Data Glove Ultra Manual v1.3.pdf, Version 1.3, January 2011.
  19. Le Gall, Jean-François. "Random trees and applications." Probab. Surv 2.245-311 (2005): 15-43.
  20. Viola, Paul, and Michael Jones. "Rapid object detection using a boosted cascade of simple features." Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. IEEE, 2001.
  21. Haykin, Simon (2004) Neural Network. "A comprehensive foundation." Neural Networks 2.2004.
  22. Cover, Thomas, and Peter Hart. "Nearest neighbor pattern classification." Information Theory, IEEE Transactions on 13.1 (1967): 21-27.
  23. Rish, Irina. "An empirical study of the naive Bayes classifier." IJCAI 2001 workshop on empirical methods in artificial intelligence. Vol. 3. No. 22. 2001.
  24. Bradski, Gary, and Adrian Kaehler. Learning OpenCV: Computer vision with the OpenCV library. " O'Reilly Media, Inc.", 2008.
  25. Müller, Meinard. "Dynamic time warping." Information retrieval for music and motion (2007): 69-84.
  26. Hess, Roland. The essential Blender: guide to 3D creation with the open source suite Blender. No Starch Press, 2007.
  27. MySQL, A. B. "MySQL." (2001).
  28. Creighton, Ryan Henson. Unity 3D Game Development by Example: A Seat-of-Your-Pants Manual for Building Fun, Groovy Little Games Quickly. Packt Publishing Ltd, 2010.
  29. Trujillo, Stan. Cutting-edge Direct3D programming: everything you need to create stunning 3D applications with Direct3D. Coriolis Group books, 1996.
  30. Claypool, Mark, Kajal Claypool, and Feissal Damaa. "The effects of frame rate and resolution on users playing First Person Shooter games." Electronic Imaging 2006. International Society for Optics and Photonics, 2006.
  31. Wegner, Peter. "Interoperability." ACM Computing Surveys (CSUR) 28.1 (1996): 285-287.
Download


Paper Citation


in Harvard Style

Escudeiro P., Escudeiro N., Reis R., Rodrigues P., Lopes J., Norberto M., Baltasar A., Barbosa M. and Bidarra J. (2015). Real Time Bidirectional Translator of Portuguese Sign Language . In Proceedings of the 11th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST, ISBN 978-989-758-106-9, pages 203-213. DOI: 10.5220/0005448502030213


in Bibtex Style

@conference{webist15,
author={Paula Escudeiro and Nuno Escudeiro and Rosa Reis and Pedro Rodrigues and Jorge Lopes and Marcelo Norberto and Ana Bela Baltasar and Maciel Barbosa and José Bidarra},
title={Real Time Bidirectional Translator of Portuguese Sign Language},
booktitle={Proceedings of the 11th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,},
year={2015},
pages={203-213},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005448502030213},
isbn={978-989-758-106-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 11th International Conference on Web Information Systems and Technologies - Volume 1: WEBIST,
TI - Real Time Bidirectional Translator of Portuguese Sign Language
SN - 978-989-758-106-9
AU - Escudeiro P.
AU - Escudeiro N.
AU - Reis R.
AU - Rodrigues P.
AU - Lopes J.
AU - Norberto M.
AU - Baltasar A.
AU - Barbosa M.
AU - Bidarra J.
PY - 2015
SP - 203
EP - 213
DO - 10.5220/0005448502030213