PRINT-BASED INTERACTION INTERFACES FOR
MULTILINGUAL MULTIMEDIA AND SIGN LANGUAGE
ELECTRONIC RESOURCE INTEGRATION
Kamen Kanev
Research Institute of Electronics, Shizuoka University, Hamamatsu 432-8011, Japan
Reneta Barneva
Department of Computer Science, State University of New York, Fredonia, NY 14063, U.S.A.
Valentin Brimkov
Mathematics Department, SUNY Buffalo State College, Buffalo, NY 14222, U.S.A.
Dimitrina Kaneva
Faculty of Education, Canterbury Christ Church University, Canterbury, KENT CT1 1QU, U.K.
Keywords: Multilingual dictionary, Sign language, Multimedia database, Learning support, Cluster Pattern Interface
CLUSPI, Print-based interactions.
Abstract: In this work we outline previous implementations of multilingual multimedia dictionaries and discuss
possibilities for adding new functionalities and expanding their coverage. Independently developed sign
language dictionary resources are further explored and considered for inclusion in an integrated multilingual
multimedia dictionary with video support. Print based interfaces for direct access to digital content are
implemented and a novel concept for dynamic linking to printed documents based on mapping of printed
and digital content is proposed. Printed texts in different languages and language independent images are
used as interface components for addressing diverse multimedia content including sign language and lip
reading multimedia resources.
1 INTRODUCTION
Despite the proliferating electronic information
sources, printed media continues to be widely
employed. Printed materials, however, do not
provide native support for audio, video and
interactive multimedia content. On the other hand,
multimedia is quite important in language
acquisition and could bring to significant
improvements, especially when young children are
involved. Multimedia content is also indispensable
for people with hearing disabilities and other
impairments. Although various printed and
multimedia materials for young and hearing
impaired learners already exist, their integrated use
is rather limited. Instruction could benefit if contents
for different learning audiences are brought together
in a supportive system for integrated use of printed
and multimedia content.
In this work we discuss how multimedia
dictionaries with multilingual word definitions,
pronunciation samples, and illustrations (Barneva et
al., 2009) could be integrated with sign speech and
sign language electronic resources (ASLPro, Sign
Language Resources, Inc.). An important aspect of
our approach is the tangible print-based interface for
accessing and interacting with the multimedia
content (Kanev and Kimura, 2005; Kanev and
Kimura, 2006). Through it, multimedia contents
targeting various audiences and controlled separately
223
Kanev K., Barneva R., Brimkov V. and Kaneva D. (2009).
PRINT-BASED INTERACTION INTERFACES FOR MULTILINGUAL MULTIMEDIA AND SIGN LANGUAGE ELECTRONIC RESOURCE INTEGRATION.
In Proceedings of the 4th International Conference on Software and Data Technologies, pages 223-228
DOI: 10.5220/0002255802230228
Copyright
c
 SciTePress

could be smoothly integrated into a unified system.
Because of the space limit, here we present only the
basic features and some possible applications of our
extensive theoretical and experimental work. A
complete presentation including all technical details
will be provided in a full-length journal version of
the paper.
2 MULTIMEDIA DICTIONARIES
Nowadays, one can use talking dictionaries, visual
(picture) dictionaries, and also combined
multimedia dictionaries. Creating such dictionaries
as well as studying and analyzing the issue from
theoretical, practical or other points of view, have
attracted a lot of interest and resources in recent
years. Even a brief review and rough classification
of existing electronic dictionaries is a difficult task.
There are monolingual dictionaries (see, e.g.,
Houghton Mifflin Interactive; Turrini et al, 2000;
Optimus Pascal, 1998; DK Multimedia), bilingual,
and multilingual dictionaries (see, e.g., Chinese-
English Multimedia Dictionary; Focus Company;
HuiFeng; ILSP).
Figure 1: An audio book tablet with a pen pointer.
For some audiences, and especially for young
children, specialized devices for simultaneous
presentation and integration of printed and
multimedia content are often employed. The audio
book tablet shown in Fig.1, for example, would
allow a printed picture book and an accompanying
sound cartridge to be installed onto it and then used
synchronously. If children touch the printed content
with the pen, the underlying position sensing device
links that to a pre-recorded sound which is played
from the installed sound cartridge.
Such audio book tablets, however, can only
accommodate printed materials in predetermined
sizes that properly fit on the tablet. Flexible scanning
tablets that employ digitally encoded layers
embedded into the printed document content and do
not require physical scanning tables can also be
employed. In this way printed documents could be
easily enhanced with digitally encoded positional
information that is practically invisible for the naked
human eye but is easily perceptible for specialized
optical reading devices (Kanev and Kimura, 2005;
Kanev et al., 2008).
3 SIGN LANGUAGES AND
DICTIONARIES
In contrast to spoken languages that employ sound
patterns, sign languages rely on visual patterns to
convey meaning. Employed visual patterns are quite
complex and manual communication, lip
movements, face expressions, and body language are
often simultaneously used. It is important to note
that due to certain historical developments no one-
to-one match of signed and spoken languages
currently exists. English speaking countries such as
the UK, the USA, and Australia, for example,
employ quite different sign language systems
(Napier, 2006). Australian sign language is related to
the UK sign language but it is also influenced by the
American and Irish sign languages. Consequently,
hearing impaired persons from those countries could
use written English, but not their sign languages, to
communicate.
Some sign languages on the other hand may
have similarities and even share signs while their
corresponding spoken languages are quite distinct
and even belong to different language groups.
Czech, Slovak, and Hungarian sign languages, for
example, are naturally clustered together (Bickford,
2005) despite the fact that spoken Hungarian is quite
unique and not related to the languages of the
neighbouring countries. Similarly, Bulgarian and
Romanian sign languages are also clustered together,
although the spoken languages of the two countries
belong to different language groups.
Different assistive methods and technologies
could help in enhancing and augmenting the
instructional process for hearing impared (Adamo-
Villani and Beni, 2004; Zamfirov et al., 2007). For
illustration we show in Fig.2 the screen interface of
a recently developed concise Japanese sign language
dictionary for English speakers. Note that the
interface and all written words are shown in English
while the videos show the signs in Japanese. In the
following sections we will discuss how the content
of this and other sign language dictionaries could be
ICSOFT 2009 - 4th International Conference on Software and Data Technologies
224

integrated with multilingual multimedia electronic
resources for better educational support.
Figure 2: Interface of a Japanese sign language dictionary
for English speakers.
4 PRINT BASED INTERFACES
FOR ACCESS TO DIGITAL
CONTENT
The system that we propose attempts to bring
together multiple information sources that can be
either online or offline. Technically, any
information on the Web can be accessed through a
standard www browser. Addressing individual
pieces of information, however, is not a trivial task
since specific links and corresponding URLs are
needed. Maintaining and organizing such links in a
convenient, easy to access way is quite time
consuming. The form in which links are presented to
the user is also very important. Nowadays, there are
many specialized magazines and newsletter entirely
dedicated to introduction of online resources. Issues
of such publications though quickly become
obsolete since they only represent snapshots of the
Web at the time of publishing and cannot grasp its
dynamic nature.
4.1 Dynamic Linking to Printed
Content
To address this problem we introduce the concept of
dynamic linking to printed materials that allows
automated matching of printed and electronic
content. When URL addresses are directly printed
on paper, dynamic linking is difficult to
implemented, since alterations of the URL addresses
are not possible. QR-code based URL linking to
www pages features similar problems. Printed QR-
codes contain directly encoded URL addresses that
are extracted and forwarded, for example, to a
browser in a mobile phone. This again makes it
difficult to change the encoded URL addresses after
they are printed on paper.
We address the above mentioned problems by
employing paper based point-and-click functionality
for implementing direct interactions with the printed
content. Through our method, printed objects of
different nature, form, and size can be associated
with system or user defined actions, such as
invoking of specific software and communicating
parameters to it. The direct interactions method
effectively eliminates the need of printed or QR-
code embedded URL links and thus allows more
flexibility in adjusting and modifying associated
actions and system responses on the fly.
Current printing technologies heavily rely on
computerized typesetting and publishers retain
digital versions of printed materials for republishing
and preparation of new editions, for placement in
digital libraries, and others. Such digital versions of
printed documents are made more widely available
by the publishers as the public demand increases and
electronic publishing becomes economically viable.
In the rare cases when electronic versions are
difficult to obtain and for experimental purposes
scans of printed materials could also be employed.
In the following presentation of the idea of
dynamic linking to printed materials, we assume that
both the printed and the digital versions of the
concerned document are available. Once printed, the
content of the paper version cannot be altered but its
digital version can. By pointing and clicking on
different printed objects we can then establish direct
mapping with their counterparts in the electronic
version of the document. When a word is printed on
paper, OCR methods need to be applied to extract
the characters comprising it. However, if access to
the same word in the corresponding digital
document is available, the comprising characters can
be directly obtained. By accessing digital versions of
printed documents, identification of printed images
and their components could also be optimized.
4.2 Mapping of Printed and Digital
Content
Our method establishes one to one mapping of a
printed and a corresponding digital document based
on their presentation layout and not on their content.
In this way adjustments in the content of the digital
version of the document could be made at any time
as long as its layout does not change. This allows
updating of hyperlinks and references to online
information sources and thus keeping the document
synchronised with the dynamic content available on
the web.
PRINT-BASED INTERACTION INTERFACES FOR MULTILINGUAL MULTIMEDIA AND SIGN LANGUAGE
ELECTRONIC RESOURCE INTEGRATION
225

(a) (b)
Figure 3: Mapping of printed and digital content.
If a user points and clicks on the printed word
“laughed” by a specialized CLUSPI reader as shown
in Fig.3a, a Unique Document Identifier (UDI),
Publisher Sheet and Page (PSP) positioning
information, and the CLUSPI reader rotation in
respect to the printed page could be instantly
obtained. The UDI is converted to a PDF file name
containing the digital version of the printed
document that can be opened with PDF reader
software and displayed as shown in Fig.3b The PSP
data identifies a publisher sheet of size up to A0+
along with a horizontal and a vertical offsets from its
top left corner. Based on this data we derive the
corresponding printed page number and the position
of the CLUSPI pointer on that page. Obtained data
can be used for instructing the PDF reader to show
the page on the screen and then the cursor can be
moved to the derived position. At that point, if a
double click with the left mouse button is
effectuated, the word under the cursor would be
automatically selected and highlighted as shown in
Fig.3b.
With this we have demonstrated how clicking on
a word in a printed document could be used for
automated extraction of its character string from a
corresponding digital document. In practice, to avoid
interference with other running applications, one or
more virtual screens may need to be used.
5 PRINT BASED ACCESS TO
MULTIMEDIA CONTENT
The PDF approach discussed so far is suitable for
printed and digital documents containing mainly text,
pictures and hyperlinks. When sound and video are
also involved, using a standard www browser would
be more convenient. In the following sections we
discuss how a print-based point-and-click interface
is employed for integration of multilingual
multimedia and sign language electronic resources
through a www browser.
5.1 Text based Printed Interfaces
Printed materials for foreign language education
may be either entirely prepared in the target foreign
language or they can also use the mother tongue for
translation and explanations. In Fig.4 we show some
typical CLUSPI enhanced content of an English
language textbook for Japanese students that we will
use as an example in our further discussion.
Figure 4: Sample CLUSPI enhanced content for Japanese
learners of English.
Students are expected to read the text, to learn
the highlighted new words and to make sure that
they fully comprehend the entire dialog. A native
speaker recording of the dialog is available on an
accompanying CD, so students can use it for
listening and pronunciation training. We have
employed the above CLUSPI enhanced content for
conducting experiments with the print based point-
and-click functionality that we have discussed
earlier. The following links between the printed
content and the audio have been established:
z English pronunciation of highlighted words
z Japanese translation of highlighted words
z Reading of sentences in English
z Translation of sentences in Japanese
Playback of voice recordings associated with the
highlighted words is initiated by pointing and
clicking on them with the CLUSPI reader as shown
in Fig.4. The CLUSPI reader orientation in respect
to the printed document is used to discriminate
between the English pronunciation and the Japanese
translation of the highlighted words. With this
interface, students can instantly access voice
recordings for new words in any order and can easily
replay them for training, as many times as needed.
Playback of recording of the dialog sentences is
initiated by pointing and clicking in the white
margin in front of each sentence. Recorded voice
readings of sentences in English are helpful for
hearing practice and for natural English intonation
training. Japanese translations, on the other hand, are
instrumental for the verification of understanding.
ICSOFT 2009 - 4th International Conference on Software and Data Technologies
226

5.2 Image based Printed Interfaces
Pictorial illustrations or pictograms are generally
considered as language neutral because they could
convey certain meaning without resorting to writing
or pronunciation. Printed illustrative materials would
therefore be suitable for building interfaces and
accessing dictionary content in multiple languages.
In Japan, for example, comics books or “manga” are
very popular and could be used as natural print-
based picture interfaces for language learning. The
comics characters could speak in different languages,
depending on the learners’ mother tongue and the
target foreign language.
A multilingual multimedia dictionary for
children containing graphical illustrations of all
dictionary entries along with pronunciation
recordings and translations in Japanese, English,
Turkish, French, and Swahili has already been
presented in (Barneva et al., 2002). We have
expanded its functionality and have developed a
print based interface (Barneva et al., 2008) for
direct interaction support. The dictionary includes
various options like grouping of words in different
categories and preview of typical sentences, spelling,
related images, and others.
There are also video clips for helping children
understand difficult notions and expressions. Word
puzzles and multilevel games further support
language learning in a natural way and contribute to
the relaxing atmosphere during the study. Quizzes
for evaluation and self-evaluation of the current
knowledge are also available.
5.3 Printed Interfaces for Sign
Languages
Entries in traditional printed sign language
dictionaries usually contain one or more pictures or
drawings of the signer accompanied with a textual
explanation. This may be good as a reminder to help
recall some signs already familiar to the signer but it
is definitely not sufficient for new signs and for self
study. Sign language dictionaries and materials for
lip reading training could obviously benefit if access
to video content is provided. We will continue,
therefore, with discussing how the multimedia base
of our dictionary could be extended to include sign
language video and what influence that might have
on the print based interface to the dictionary content.
Some typical entries of a printed American Sign
Language (ASL) dictionary describing how to sign
the numbers 16, 17, 18, and 19 are shown in Fig.5.
Both, the textual part and the illustrative parts of the
entries, could be linked to instructional videos. In
this case linking of multiple videos introducing the
sign from different perspectives and by different
persons may be helpful for improving understanding
and retention. To introduce the number signs in
different sign languages, however, new printed
content would be needed, since the one in Fig.5 is
bound to ASL.
Illustrativedrawingsforsigningthenumbersfrom16to19
Tex t u a l descriptionforsigningthenumbersfrom16to19
Figure 5: Sample number entries in a printed sign
language dictionary.
For a successful integration of video support in a
multilingual sign language dictionary, therefore, the
printed contents need to be carefully redesigned.
Instead of illustrating how to sign a word in a
specific sign language, the printed contents should
include drawings or pictograms capturing and
conveying the meaning of the word. This is exactly
what we have attempted to achieve with the printed
iconic interface to our previously developed
multilingual multimedia dictionary for children. We
take this idea one step further and consider
introducing the word signs not individually, but in
the context of short sentences and popular stories
(Fig.6). In this way, meanings of target individual
words become clearer and easier to grasp by the sign
language learners.
We are currently using the Aesop’s fable “The
Fox and the Grapes” for experimenting with screen
based and print based interfaces. The screen
interface shown in Fig.6 is accessed through a
standard browser where clicking on the pictures
invokes the multimedia dictionary entry
presentations in a separate window. The page
content shown in the browser can be printed on
CLUSPI encoded paper and then used as interface
control component as explained earlier.
PRINT-BASED INTERACTION INTERFACES FOR MULTILINGUAL MULTIMEDIA AND SIGN LANGUAGE
ELECTRONIC RESOURCE INTEGRATION
227

Figure 6: Aesop’s fable “The Fox and the Grapes”
employed as multilingual multimedia dictionary interface.
6 CONCLUSIONS
In this work we have introduced our approach for
providing support for children and hearing impaired
persons through multilingual multimedia
dictionaries employing video, and screen- and print-
based interfaces. We are planning to continue with
adding more sign language and lip reading resources
from different languages, expanding the story
coverage, and providing support for different
educational games and other activities.
ACKNOWLEDGEMENTS
We express our gratitude to Andras Zentai for his
involvement in the development of the experimental
software and the preparation of the sample materials.
REFERENCES
Adamo-Villani, N., Beni, G., 2004. Automated Finger
Spelling by Highly Realistic 3D Animation, British
Journal of Educational Technology, Vol. 35, No. 3, pp.
345-362, 2004.
ASLPro: American Sign Language Video Dictionary.
Available online at: http://www.aslpro.com/
Barneva, R.P., V.E. Brimkov, P.L. Stanchev, 2002.
Multilingual Multimedia Electronic Dictionary for
Children, In: SSGRR 2002 – International Conference
on Advances in Infrastructure for Electronic Business,
Science, and Education on the Internet, L'Aquila, Italy,
2002, Paper No 65, 6 pages.
Barneva, R. P., Brimkov, V. E., Kanev, K. D., 2008.
Electronic Multimedia Dictionary with Direct-Access
Printed Interface, In: Barneva, R. P., Brimkov V. E.,
(Eds.), Image Analysis: From Theory to Applications,
Research Publishing Services, Singapore (2008),
39-48.
Barneva, R., Brimkov, V., Kanev, K., 2009. Combining
Ubiquitous Direction-Sensitive Digitizing with a
Multimedia Electronic Dictionary for Enhanced
Understanding, Int. Journal of Imaging Systems and
Technology, Vol. 19, No. 2, June 2009, pp. 39-49.
Bickford, J. A., 2005. The Signed Languages of Eastern
Europe, SIL Electronic Survey Reports, SIL
International and University of North Dakota, 2005,
Available online at: http://www.sil.org/silesr/
abstract.asp?ref=2005-026
Chinese-English Multimedia Dictionary, Available online
at: http://www.chinasprout.com/shop/S023.
DK Multimedia, My first incredible, amazing dictionary,
Available online at: http://www.gslis.utexas.edu/
~kidnet/reviews/dictionr.html.
Focus Company, Junior talking dictionary, Available
online at: http://www.pmjcomputers.com/software/
children/ESS062.html.
Houghton Mifflin Interactive, The American heritage
children's dictionary, Available online at: http://
www.houghtonmifflinbooks.com/catalog/titledetail.cf
m?titleNumber=J37606
HuiFeng, HuiFeng children's picture dictionary, Available
online at: http://www.asti-global.com/brochures/
Hfmmv11a.pdf.
ILSP, Lexipedia, Available online at: http://www.ilsp.gr/
lexipedia_eng.html.
K. Kanev and S. Kimura, 2005. Digital information
carrier, Patent Reg. No 3635374, Japan Patent Office,
2005.
Kanev, K., Kimura, S., 2006. Direct Point-and-Click
Functionality for Printed Materials, The Journal of
Three Dimensional Images, Vol. 20, No. 2, 2006, pp.
51-59.
Kanev, K., Morishima, Y., Watanabe, K., 2008. Surface
Code Readers for Image Based Human-Computer
Interfaces, In Proceedings of the Eleventh Int. Conf.
on Humans and Computers HC’08, Nagaoka, Japan,
November 20-23, 2008, pp. 57-62.
Napier, J., 2006. Effectively Teaching Discourse to Sign
language Interpreting Students, 2006. Language,
Culture and Curriculum, Vol. 19, No. 3, 2006, pp.
251-126.
Optimus Pascal, 1998. Polish picture dictionary for
children, Bielsko-Biala, 1998, Available online at:
http://www.polonia.com/polishbooks/product.asp?sku
=4295.
Sign Language Resources, Inc., Available online at:
http://www.signlanguageresourcesinc.com/
Turrini, G., Paccosi, A., Cignoni, L., 2000. Combining the
children's dictionary Addizionario with a multimedia
Activity Book, Proc. Ninth Euralex International
Congress, Stuttgart, 2000, pp. 107–112.
Zamfirov, M., Saeva, S., Popov, T., 2007. Innovation in
teaching deaf students physics and astronomy in
Bulgaria, Physics Education, Vol. 42, No. 1, January
2007, pp. 98-104.
ICSOFT 2009 - 4th International Conference on Software and Data Technologies
228