![](bg3.png)
the application, the application was configured to use
the Dutch version of Android’s speech recognition.
For billboards, menus in restaurants, information
signs, and other written texts that users may encounter
during their daily-life, the fourth option may be used.
Users can take a picture of the incomprehensible text
using the camera of their mobile device. The software
automatically focusses the image and subsequently
sends it to an OCR (Optical Character Recognition)
service which converts the image into readable text.
Next, the recognized text can be sent to the server for
analysis or users can opt for taking a new picture if the
OCR was not successful due to e.g. a blurry picture.
In our application, the OCR software of WiseTrend is
used (WiseTREND, 2011), but alternative solutions
are easily to integrate. This OCR service is avail-
able for different languages (we used the Dutch ver-
sion for testing the application) and provides features
like deskewing, removing texture, automatic rotation
of the picture, and even detecting barcodes.
If deaf children are experiencing difficulties to un-
derstand a received SMS message, the fifth option can
be used. An SMS message or a part of the text of the
SMS can be selected for transmission to the server
that analyzes that piece of text. Similar, (an extract
of) an e-mail can be selected and sent for clarification
to the server via the last input option.
3.2 Sentence Analysis
An important aspect of the analysis of the sentences
that users submit is tracing the most important words,
and thereby the context of the sentence. Different
solutions are possible to detect the most important
words of a sentence: using dictionaries to get the word
class, part-of-speech taggers that get the word class
in a more intelligent way by analyzing the sentence,
frequency tables that provide information about how
many times a word is used in the language, and the-
sauri that contain a set of related words (such as syn-
onyms, hyponyms, and antonyms).
The current implementation of our conversation
assistant uses a dictionary and a part-of-speech tag-
ger. Frequency tables and a thesaurus are features for
future versions of the application to improve the text
analysis. For the English version of the conversation
assistant, the English version of WordNet can be used
(this is a lexical database developed at the University
of Princeton). However, because of the license fee of
the Dutch version, WordNet was not incorporated in
the current version of the conversation assistant that
was evaluated by potential end-users.
OpenTaal is a project about spelling, hyphenation,
thesauri, and grammar for the Dutch language (Open-
taal, 2011). By harvesting sentences from govern-
ment websites and online newspapers, the project
has gathered a database with speech information,
examples, and derivations for about 130,000 Dutch
words. Although this database is not yet publicly
available, the contributors of the OpenTaal project put
the database at our disposal for the retrieval of speech
and word information in the conversation assistant.
Based on this database, it is possible to configure
which types of words have to be retrieved from the en-
tered text (e.g., nouns and verbs can be used to search
for explanatory pictures or videos). For performance
reasons, the database is locally stored on the device of
the end-user in the current implementation, but an on-
line version is also possible. Regrettably, some words
can have a different interpretation and word class de-
pending on the sentence in which the word is men-
tioned. E.g., “minor” in the sentence “Kids under 18
are considered minors” is a noun, whereas in the sen-
tence “I had a minor accident” minor is an adjective
and has a different meaning.
This difficulty is solved by utilizing a part-of-
speech tagger, a technique which assigns attributes to
the words of a sentence and derives the word class
based on information from the entire sentence. The
part-of-speech tagger ‘Frog’, which is used for this
purpose, provides more accurate results than the anal-
ysis based on merely the database of OpenTaal. Frog
is an integration of memory-based natural language
processing (NLP) modules developed for the Dutch
language. Frog’s current version will tokenize, tag,
lemmatize, and morphologically segment word to-
kens in Dutch text files, and will assign a dependency
graph to each sentence (Frog, 2011). Because of sev-
eral software dependencies, we opted to deploy the
software on a server and send requests for analyzing
sentences from the mobile device.
3.3 Filtering and Watching Multimedia
After analyzing the text and filtering out unimportant
words, the remaining words or groups of words are
used to query various content sources via publicly-
available APIs. In the current implementation, dif-
ferent sources are used to find images and videos:
Google Image Search, Flickr, YouTube, and Vimeo.
Moreover, new content sources, filters and hint-
generating systems (which are explained later) can
easily been integrated because of the build-in plugin
structure of the application. These content sources are
evaluated and receive a rating to determine the order
of the results for subsequent queries. This rating con-
sists of two elements: an explicit score that the user
has set in the settings menu and an implicit score de-
AMOBILECONVERSATIONASSISTANTTOENHANCECOMMUNICATIONSFORHEARING-IMPAIRED
CHILDREN
777