Authors:
Paula Escudeiro
1
;
Nuno Escudeiro
1
;
Rosa Reis
1
;
Pedro Rodrigues
1
;
Jorge Lopes
1
;
Marcelo Norberto
1
;
Ana Bela Baltasar
1
;
Maciel Barbosa
2
and
José Bidarra
3
Affiliations:
1
Instituto Superior de Engenharia do Porto, Portugal
;
2
Faculdade de Engenharia da Universidade do Porto, Portugal
;
3
Universidade Aberta, Portugal
Keyword(s):
Portuguese Sign Language, Education, Deaf, Kinect, Sensor Gloves, Translator, Machine Learning, OpenCv, Support Vector Machines, Dynamic Time Warping, Avatar, Serious Games.
Related
Ontology
Subjects/Areas/Topics:
Multimedia and User Interfaces
;
Web Information Systems and Technologies
;
Web Interfaces and Applications
Abstract:
The communication with deaf by means of written text is not as efficient as it might seem. In fact, there is a
very deep gap between sign language and spoken/written language. The deployment of tools to assist the daily
communication between deaf people and the rest may be a significant contribution to the social inclusion of
the deaf community. The work described in this paper addresses the development of a bidirectional translator
between Portuguese Sign Language and Portuguese text and a serious game to promote the learning of the
Portuguese Sign Language. The translator from sign language to text employs two devices, namely the
Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands.
The hands configurations are classified using Support Vector Machines. The classification of the movement
and orientation of the hands is achieved through the use of Dynamic Time Warping algorithm. The translator
exhibits a precision higher than 90%. In
the other direction, the translation of Portuguese text to Portuguese
Sign Language is supported by a 3D avatar which interprets the entered text and performs the corresponding
animations. As a complement, we also present a serious game directed to assist in the difficult task of learning
the Portuguese Sign Language.
(More)