A Mobile Augmentative and Alternative Communication (MAAC)
Application for Disabilities
Ka-Lun Cheung, Toby H. W. Lam and King-Hong Cheung
Department of Computing, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong
Keywords: Human-machine Interfaces for Disabled People, Development of Assistive Technology.
Abstract: Communication board is one of Augmentative and Alternative Communication (AAC) which provides a
large amount of pictures classified under various categories. It is a useful instrument for disabled people,
who are suffering from speech and language impairment, to facilitate their communication by simply
selecting pictures. The traditional communication board would assist the users to communicate with the
others. However, there are a number of limitations of the communication board such as large in size and
difficult to carry. Because of the advanced technology, it is inexpensive to purchase a mobile device. In this
paper, we present our mobile augmentative and alternative communication application, namely MAAC app,
for disabled people. The MAAC app runs on the Android-based mobile devices such as smartphones or
tablets. This application is suitable for disabled people who are suffered from cerebral palsy, spasm and
stroke. We performed user acceptance test (UAT) with domain experts on the application and the overall
comment is positive.
1 INTRODUCTION
According to the World Report on Disability (World
Health Organization, 2011), there is about 15% of
the world’s population were estimated to be living
with disability. It is a large amount of population
that we should not ignore. The disabled people may
face different kinds of challenges in their daily life.
One of the major challenges is communication. In
general, the common communication approach
between disabled and normal person includes: (1)
sign language, (2) lip-reading, (3) writing and (4)
body language. However, those mentioned methods
are not suitable for all types of disabled person,
especially cerebral palsy patients, spasm patients and
physical handicapped. Thus, it may introduce barrier
for those patients in communication and they may
encounter problem in expressing themselves.
In this paper, we illustrate how we designed a
mobile augmentative and alternative communication
application, namely MAAC app, for disable people.
The MAAC app runs on Android-based mobile
devices such as smartphones and tablets. By using
this app, different degree of disabilities would utilize
it in communication. To ensure the developed
mobile application really assist the disabled people
in communication, we collaborated with the
Association for Engineering and Medical Volunteer
Service (EMV 2013) to go through the whole
development cycle – from user requirements
collection to user acceptance testing. EMV is a non-
governmental, non-profit organization in Hong
Kong SAR. The goal of EMV is to enhance the
independence and quality of life of people with
disabilities and elderly through professional
voluntary services. They provide a number of
services for the disabilities and elderlies such as
rehabilitation aids services, computer rehabilitation
services, vocational rehabilitation services,
community rehabilitation services, elderly services
and social enterprise. EMV is an organization with
lots of professionals from different domains such as
engineering, medical and paramedical and also the
EMV staffs have great experience in communicating
with disable people. They understand the needs of
the disabled person. Furthermore, their professional
comments and advices have been valuable to us in
developing a mobile application which fits the
requirements of the disabled people.
The main contributions of our proposed mobile
augmentative and alternative communication
application (MAAC) are: (a) providing a user
friendly communication tool for disabled persons, (b)
188
Cheung K., H. W. Lam T. and Cheung K..
A Mobile Augmentative and Alternative Communication (MAAC) Application for Disabilities.
DOI: 10.5220/0004738201880195
In Proceedings of the International Conference on Health Informatics (HEALTHINF-2014), pages 188-195
ISBN: 978-989-758-010-9
Copyright
c
2014 SCITEPRESS (Science and Technology Publications, Lda.)
enhancing the convenience of usage with
information technology, (c) providing language
localized content to target users in the Hong Kong
SAR and (d) the application can be customized so
that different level of disable people would use it in
communication.
The rest of this paper is organized as follows:
Section 2 describes the literature review. Section 3
describes the system design and implementation.
Section 4 presents the results of User Acceptance
Test (UAT). Section 5 offers our conclusion.
2 LITERATURE REVIEW
Augmentative and Alternative Communication
(AAC) is about the communication methods for
people who are suffering of speech and language
impairments in substituting speaking or writing
(Beukelman and Mirenda, 2005; Schlosser 2003). It
is widely adopted by people who have been
diagnosed in speech and language impairments such
as cerebral palsy and autism. The main aim of the
communication board is to “arrange language in
space so individuals can, by selecting from the
available options, say what they wish to say as
quickly as possible, and can do so with a minimal
amount of effort” (Blackstone, 1993).
Picture Exchange Communication System
(PECS) or Picture Exchange (PE) is a type of AAC
modes (Bondy and Frost, 2001). Using PECS, user
can pick symbol cards from the card set and show
his/her message to their target audience for
communication. One of the PECS is communication
board/book. Communication board is a picturebased
tool of aided AAC, which is about using a device for
exchanging messages. It lets user to express or
communicate by pointing or looking at a particular
word, symbol, picture or photo listed on the
communication board. In general, communication
board is displayed in grid form, with different words
and phrases shown in the grid. The words are mainly
grouped under categories, such as food, places and
activities. Using communication board, the
communication efficiency and effectiveness of
people who is suffering with speech and language
impairments could be improved. However, one of
the limitations of the traditional communication
board is in large size. Thus, it reduces the level of
portability.
Sono Flex (Sono Flex, 2013) is a commercial
mobile application developed by Tobii Technology.
This application serves as an AAC vocabulary tool
with English user interface only. They developed the
application which supports Apple iOS, Android OS,
Amazon Kindle Fire and Tobii Communicator. Sono
Flex displays many words, phrases or short
sentences on the screen. Also, the application allows
user to categorize types of words. For each
word/phrase, the application would also show the
related image in the button. Once the user forms a
sentence, the application can pronounce it to the
others. However, Sono Flex does not allow user to
add any new word. This reduces the flexibility in
updating its word/phrase library. Furthermore, Sono
Flex provides English user interface only which
makes it difficult for local users in the Hong Kong
SAR to adopt it for communication.
Com Aid (Com Aid, 2013) is a free Android OS
mobile application which serves as a communication
board. The application, which was developed by
Tung Wah Group of Hospitals Kwan Fong Kai Chi
School in the Hong Kong SAR, provides Traditional
Chinese interface and it supports text to speech
conversion. It assists the users with speech disorder
to communicate and hence to encourage the users to
communicate with others. Com Aid offers a
colourful user interface. Similar to Sono Flex, it
displays the word with related image. This
application lets users to add new word/phrase with
the picture and pronounce the sentence. However,
sometimes it is difficult to handle drag and drop the
word/phrase to form a sentence when using Com Aid.
Also, if the user selects a different word/phrase
category, the application does not update the screen
to reflect the change. Thus, user cannot select the
word/phrase properly and it may introduce barrier for
the disabled user to use it for communication.
3 SYSTEM DESIGN AND
IMPLEMENTATION
3.1 Limitations in Existing AAC
Mobile Applications
The traditional communication board is typically
large in size and limited in portability. Currently,
because of the advanced development in technology,
it is inexpensive to purchase mobile devices such as
smartphone or tablet. At the moment, there are quite
a number of AAC applications for mobile devices in
the market. Nonetheless, we cannot find an AAC
application that the content and user interface of the
application could be localized, e.g. Traditional
Chinese for Hong Kong SAR users. If the mobile
application only provides user interface in English, it
AMobileAugmentativeandAlternativeCommunication(MAAC)ApplicationforDisabilities
189
would limit the Hong Kong disabled users to use
those AAC applications in their daily
communication. Furthermore, we discovered that the
existing AAC mobile applications are not flexible
enough to allow user, his/her friends, family
members, caregivers or the therapists to do any
customization. Due to these aforementioned
limitations in the existing AAC mobile applications,
we proposed to develop a new mobile augmentative
and alternative communication application, namely
MAAC app. The MAAC app aims to:
(1) Provide an accessibility and localized tool for
assisting disabled users, in Hong Kong SAR, to
communicate with the others,
(2) Be easy to use by different degree of disabilities,
(3) Be able to customize, by the user or his/her
friends, family members, caregivers or the
therapists, so as to match with the degree of the
disabilities of the user.
3.2 Requirements
We held more than 10 meetings with EMV staff to
collect the user requirements. We collected not only
the functional requirements but also nonfunctional
requirements. Furthermore, during the meetings, we
discussed a number of user interface design issues
that need to be handled during the development.
Since the application is mainly designed for disabled
users, some of the requirements on the user interface
design, such as high contrast text, resizable text and
large buttons, and functions, such as assisted auto-
scan mode, are tailor made for disabled users. The
functional and non-functional requirements are
presented in Section 3.2.1 and 3.2.2 respectively.
Section 3.3 presents the System architecture of the
MAAC app. Section 3.4 presents the design issues
of the MAAC app.
3.2.1 Functional Requirements
There are two types of major users in using the
MAAC app: (a) the disabled people and (b) the
normal people such as the users’ family members,
friends, caregivers or therapists. The functions of the
MAAC app would be divided into two aspects: (1)
general usage and (2) customizable settings. The
general usage is to use the MAAC app to compose
phrase/sentence by pressing the buttons in the main
page interface. Once the user composed a
phrase/sentence, the play button can be pressed to
pronounce the phrase/sentence using pre-recorded
sound clip or online speech synthesizer. With this
feature equipped, the user can be more effective in
communicating with others. The customizable
settings let the disabled user to have control in a
number of items of the application such as the font
size and words sorting mode. The detail of the
customizable settings is given in Section 3.4.4. If the
disabled user is not able to change the settings by
himself/herself, he/she can ask a normal person (e.g.
his/her friends, family members, caregivers or
therapists) for help in changing the settings of the
MAAC app. However, if the degree of disabilities of
the users allows them to control the user interface
properly, they can also change the customizable
settings by themselves. Figure 1 shows the use-case
diagram of the MAAC app.
Figure 1: Use case diagram of the MAAC app.
One of the objectives in developing MAAC app
is to provide a well-designed user interface for the
users so that they can compose phrases/sentences
conveying their message and to be read out to the
message recipients easily. The following is a list of
functionalities that MAAC app provides:
Manage word/phrase (with corresponding
picture and sound clip) database – MAAC app
allows user to add in new word/phrase with
optionally the desired corresponding picture
and sound clip for pronouncing the
word/phrase or changing the existing entries in
the word/phrase database. Thus, it lets the user
able to create his/her own personal AAC.
HEALTHINF2014-InternationalConferenceonHealthInformatics
190
Capture/Select a photo for a word/phrase –
MAAC app allows user to take a photo with
the built-in camera as one source of providing
the desired picture for a word/phrase, which is
shown as the background image of the button
for select that word/phrase. Alternatively, a
photo in the photo library of the device can be
selected serving the same purpose. With this
feature, it provides an alternative, possibly
easier, way to remember or recognize the
(writing of) word/phrase and therapist can use
the image to train the user how to effectively
communicate with the others (Bondy and Frost,
2001). This would further enhance the
disabled user’s motivation in communication.
Read out a phrase/sentence – after the user has
composed a phrase/sentence, he/she can press
the “Play” button to pronounce the sentence
using the online speech synthesizer or a pre-
recorded sound clip.
Record/Select a sound clip for a word/phrase –
MAAC app allows users using their own voice
in recording a sound clip for each word with
the device's voice recording function, i.e.
MAAC app can use the user's own voice
recorded sound clip to pronounce the
word/phrase. Alternatively, a sound clip can
be selected from the device's media library.
3.2.2 Non-functional Requirements
There are some nonfunctional requirements for the
MAAC app.
Accessibility: the application should be easy to
use on Android-based mobile devices. The design
of the interface should be able to fit and run in
different screen resolution of different devices.
Usability: the application and its user interface
should be easy to use and understand. The
provided word/phrase is localized which related
to the daily life in Hong Kong SAR.
Interface: the application should provide
consistent user interface, be userfriendly and
simple so as to prevent any difficulty imposed on
disabled users, e.g. the button should be easily
pressed; the user interface should be in
Traditional Chinese for local Hong Kong SAR
users.
3.3 System Architecture
The MAAC app was developed using Java and the
application package file was 786k. Since the design
of the application is flexible that user can add/delete
words in the application, if the user want to add
more words, it needs more disk space from the
device. For each new word, it needs around 80KB
from the disc memory.Users are able to install and
run the MAAC app on Android-based mobile device
with Android OS version 4.0 or above. The
application is a standalone application that would
run without connection to the Internet. However, if
the user want to download a sound clip for a
word/phrase from an online source, it is necessary to
connect to the Internet via wireless or mobile
network access point. As mentioned in the previous
section, the user can also record the sound clip using
his/her voice. In case the user is not able to speak,
the user can also seek help from a normal person
(such as therapists and caregivers) to record the
pronunciation of the word/phrase. Thus, the user still
can use the pronounce sentence function even there
is no Internet connection.
Figure 2 and 3 show the entity-relationship (ER)
diagram of the word/phrase database and the system
architecture of the MAAC app respectively.
Figure 2: Entity-relationship diagram of the MAAC app.
Figure 3: System architecture of the MAAC app.
AMobileAugmentativeandAlternativeCommunication(MAAC)ApplicationforDisabilities
191
3.4 Design Issues
3.4.1 User Interface
According to the non-functional requirements, the
application should be easy to use and could be run in
different type of the Android-based mobile devices.
One of the requirements is the user’s Android device
should run Android OS version 4.0 or above. The
user interface of the application can adjust to fit
different resolution. After discussed with the EMV
professionals, we designed an interface (see Figure 4)
with serious considerations in the accessibility. The
words/phrases are grouped into six categories: Noun
(名詞), Verb (動詞), Adjective (形容詞), Adverb
(副詞), Quantifier (量詞), Time (時間). The button
of each word/phrase category has different colored
background and border for easier differentiation.
Each word/phrase is shown as a round-cornered
button, with the same border color as their
corresponding word/phrase category, in a 4x6 grid.
The text shown in button is designed with high
contrast such that the user can easily distinguish the
category and the word/phrase. Furthermore, the font
size within the button is customizable in settings (設
定). The detail of customizable settings is given in
Section 3.4.4.
Figure 4: The main user interface of the MAAC app.
First row provides six word/phrase categories (indicated
by 'a'; from left to right): Noun, Verb, Adjective, Adverb,
Quantifier and Time.
When the user presses a specific category button,
the 4x6 grid will update immediately and show the
words/phrases in the selected category. Figure 5
shows the interface when the user pressed the Time
category.
3.4.2 Normal Mode
According to the requirements, the application
should offer different input methods to serve
different disability degree of the disabled user. Thus,
Figure 5: The interface after the user pressed Time
category button.
Figure 6: (a) The text field for displaying the
phrase/sentence composed;
(b) Play button for pronouncing the phrase/sentence.
we designed two approaches to let the user to make
the selection: (1) normal mode and (2) assisted auto-
scan mode.
We designed normal mode for users who can
perform multi-touch gesture. When the user presses
a word/phrase button, the word/phrase will be
displayed in the text field (see Figure 6 – region (a)).
For example, if the user wants to compose a
sentence “I (我) want to (想) sleep (訓覺)”, he/she
presses the buttons of “I (我)”, “want to (想)” and
“sleep (訓覺)” respectively to compose the sentence
displayed in the text field. Once finished, the user
can press the "Play" button to read out the sentence.
3.4.3 Assisted Auto-scan Mode
For the assisted auto-scan mode, we target this mode
for the user with higher degree of disabilities that the
user cannot provide multi-touch gesture as input, but
he/she can touch or tap the touch screen. There are
many ways to implement the assisted auto-scan
mode. EMV professionals suggested the main screen
would be divided like a pyramid: the first level: left
and right (Figure 7 – label 1 border); once the user
make the choice by tapping the screen, the second
HEALTHINF2014-InternationalConferenceonHealthInformatics
192
level: top and down (Figure 7 – label 2 border); once
the user make the choice by tapping the screen, the
third level: column by column (Figure 7 –label 3
border). For each level, the group of buttons is
highlighted and the user can tap the screen to go into
the sub-level. However, we did not adopt this design
as this approach is not user friendly and efficient
enough. For example, if the user would like to press
the button at top row and leftmost column, the user
taps the screen 4 times.
Figure 7: Initial design of assisted auto-scan mode.
To further improve the user experience and
reduce the number of tap, we design another
approach for the assisted auto-scan mode. This
approach is illustrated in Figure 8. The group of
buttons is highlighted in column-based in a round-
robin approach (along horizontal direction; see
Figure 8 – arrow 1). The user can tap on touch
screen if he/she chooses a specific column and then
the highlight will move in row-based, again, in
round-robin approach (along vertical direction); until
the user tap the touch screen to select a specific
button (see Figure 8 – arrow 2). If the target
word/phrase does not exist in the current page, this
design also supports scanning the "Previous page"
and "Next page" buttons. When the highlight goes to
the "Next page" button, the user can tap the screen,
the word/phrase on next page will be shown. If the
user wants to select the button in the first column
and the first row, by using this approach, he/she only
needs to touch the screen 2 times. Compared with
the initial design, it reduced the number of taps
which is usually difficult for this type of target users.
However, refer to the final design, user may
spend more time in locating a specific button if the
button is located at the rightmost column and bottom
row in the grid. Since there is a time elapse in
moving from one column/row to the next
column/row, for the experienced user, he/she may
feel annoying if the scanning time between
Figure 8: Final design of assisted auto-scan mode.
columns/rows is too long. To further enhance the
user experience, it is better to provide a mechanism
to increase/decrease the scanning time interval.
3.4.4 Customizable Settings
During the design of the application, we considered
that MAAC app should be customizable to provide
personalized experience that fits the needs of
different users. After the discussion with EMV
professionals, we decided to provide the followings
in customizable settings:
a) Sorting control – the user can customize the
sorting of frequently used words/phases based
on (1) current GPS location, (2) Day-of-the-
week and (3) current time;
b) Font size control – the user can adjust the
button font size;
c) Scanning control – the user can adjust (1) the
idle time before escaping the assisted auto-
scan mode and (2) auto-scanning speed;
d) Time lag control for speech synthesis – the
user can edit the time lag of the speech
synthesizer in reading each word/phrase
Figure 9 shows the customizable settings page of
the MAAC app.
3.4.5 Insert New Word/Phrase and Category
As mentioned in Section 3.2, one of the
requirements is letting the user to insert new
word/phrase to the word/phrase database. If the user
is not able to control the application easily, he/she
can seek help from his/her friends, family members,
caregiver or therapists to insert the word/phrase.
First, the user go to the Insert new word/phrase page
(see Figure 10). At the beginning of the insert, the
buttons related to sound are disabled since a new
word/phrase does not exist at that moment.
The left-bottom area is “photo taking” area. User
can take photo which matches the word/phrase using
AMobileAugmentativeandAlternativeCommunication(MAAC)ApplicationforDisabilities
193
Figure 9: Customizable settings page in MAAC app. (a)
Sorting control; (b) Font size control; (c) Scanning
control; (d) Time lag control for speech synthesis.
the device’s camera. If the user does that, captured
photo will be underlay in the word/phrase button as
the background. After the user finished the input, the
record button and the speech synthesizer button are
then enabled. User can select which category of the
word/phrase belongs and control whether showing
the text and the photo in the button of a word/phrase.
To finish the insertion, user can press “Save” button
and then the application would check the correctness
of the input data. If there is an error, an error
message will be shown; otherwise, the application
would store all input data into the database. If the
user wants to insert a new word/phrase category,
user should have input the name of this category
first (see Figure 10 – area (a)); otherwise, when user
presses “Add” button, the application could not
insert the word/phrase category to database, and
display an error message in the screen. After input
the name, user can select the target category in the
pop up list, and user could press “Add” button to
complete the process.
Figure 10: Insert new word/phrase interface. (a) Insert new
category.
4 RESULTS OF USER
ACCEPTANCE TEST (UAT)
After finished the development, we have arranged a
User Acceptance Test (UAT) with the EMV staff
which was held on 08 May 2013 in the Jockey Club
Digital Inclusion Center (JCDIC, 2013). The
participants included occupational therapists,
engineers and professional workers of AAC. The
testing cases included: (a) layout and accessibility,
(b) basic functions and (c) customizable settings
functions.
For the layout accessibility, the EMV staff
commented that the design of application layout is
good and simple for the target users. The adopted
color scheme is easy for the target users to
distinguish the word/phrase categories. The EMV
staff provided a positive feedback on the basic
functions in MAAC app.
In UAT, the EMV staff provided a number of
feedbacks on the current user interface design.
Firstly, when the user presses the “Play” button, the
application will pronounce each word/phrase, in the
sentence, one by one. In the current design, the
pronounced word/phrase will be removed
immediately from the text field after the application
pronounced it. The EMV staff commented that all
the wordings of the sentence should keep in the text
field since the other listeners can read the sentence
from the screen and listen at the same time. In
addition, the user can re-play the same sentence.
Secondly, in the current design of the assisted
auto-scan mode, the “Play” button is scanned in the
main page only. If the user selects to browse a
specific word/phrase category, the “Play” button will
be not scanned. To be more consistent, the EMV
staff suggested the “Play” button should be scanned
in any page.
In summary, the comments from the
professional the EMV staff are positive. The
application is easy to use, fulfills the needs of the
disabled people and the content is localized and
relevant to the Hong Kong SAR.
Figure 11: User acceptance test with EMV staff.
HEALTHINF2014-InternationalConferenceonHealthInformatics
194
5 CONCLUSIONS
In this paper, we presented the details of a mobile
augmented and alternative communication
application – MAAC app. The application can be
run on Android-based mobile device such as mobile
phone or tablet. MAAC app provides a better user
experience for the disabled people. The application
was verified and evaluated by professionals such as
occupational therapists and engineers. The
application is now under final revision and EMV has
planned to release this application in Google Play
Store at the end of 2013. Users can download this
application for free.
In summary, we developed a mobile augmented
and alternative communication application for
disabled people. We fulfilled the application
requirements and finished the application with a
number of features such as customizable settings.
Nevertheless, there are still rooms for further
improvements. In the current design, the user
interface can only provide Traditional Chinese. To
let more users adopt and utilize this mobile
application, in the future, we would like to enhance
the application with a multi-lingual interface, such
as English and Simplified Chinese. Thus, the
community of the disabilities can enjoy this tool for
communication. We hope this application would
help the disabled people in communication, enhance
the digital inclusion and establish an information
society without discrimination.
ACKNOWLEDGEMENTS
We would like to thank EMV for their time and
constructive comments during the whole
development of the application.
REFERENCES
Beukelman, D., Mirenda, P., 2005. Augmentative and
alternative communication: Supporting children and
adults with complex communication needs, Baltimore:
Paul H. Brookes Publishing Co., 3
rd
edition.
Blackstone, S., 1993. Thinking a little harder about
communication displays,
Augmentative
Communication News, vol. 6, no. 1
.
Bondy, A., Frost, L., 2001. The picture exchange
communication system. Behavior Modification, vol.
25, no. 5, 725–744.
Com Aid, 2013. Online: https://play.google.com/
store/apps/details?id=com.sillycube.twkfkcs&hl=en_G
B. (last assessed in September 2013).
EMV, 2013. Online: http://www.emv.org.hk (last assessed
in September 2013).
JCDIC, 2013. Online: http://jcdic.hk (last assessed in
September 2013).
Schlosser, R 2003, The efficacy of augmentative and
alternative communication: Toward evidence-based
practice, San Diego: Academic.
Sono Flex, 2013. Online: https://play.google.com/
store/apps/details?id=com.tobii.sonoflex (last accessed
in September 2013).
World Health Organization, 2011. World report on
disability, World Health Organization.
AMobileAugmentativeandAlternativeCommunication(MAAC)ApplicationforDisabilities
195
