Gesture-Based Communication for People with Aphasia While in Bed

abio Nunes

, Ana Patr

ıcia Rocha

, Ana Rita S. Valente

, Samuel Silva

and Ant

onio Teixeira

Institute of Electronics and Informatics Engineering of Aveiro (IEETA), Department of Electronics, Telecommunications

and Informatics, Intelligent Systems Associate Laboratory (LASI), University of Aveiro, Aveiro, Portugal

Keywords:

Remote Communication, Aphasia, Accessibility, Gestures, in-Bed Scenario, Smart Environments.

Abstract:

Communication is an essential part of life and, when affected, as it happens to persons with Aphasia (PWAs),

it severely impacts their quality of life. Augmentative and Alternative Communication (AAC) approaches aim

to aid people with their communication disabilities. However, not all parts of the day are easily covered by

these solutions, e.g., when lying in bed without immediate access to them or in a state of distress that precludes

reaching for one. To address this challenge, working with PWAs is key, but the communication issues affecting

them might work as a barrier to obtain their contribution during the conceptual stage. To gather contributions

that can work as a common ground to obtain PWAs’ feedback we have been collaborating with Speech and

Language Therapists (SLTs) to propose a system that allows two-way remote communication between a PWA

lying in bed, alone, and other people (e.g., caregiver). Here, we describe this ﬁrst stage with the SLTs leading

to the conceptualization of a solution consisting of communication mediated by an assistant based on simple

Yes/No questions presented audiovisually to the PWA and answered through gestures. Overall, the SLTs

emphasized its adequateness and its strong potential to increase PWAs’ independence.

1 INTRODUCTION

Aphasia is an acquired communication impairment,

which can affect different language skills, such as ver-

bal expression and comprehension, as well as writing

and reading (Sheppard and Sebastian, 2021), having

been identiﬁed in around one-third of people after suf-

fering some type of strokes (Flowers et al., 2016).

Since verbal communication plays a pivotal role in

everyone’s daily life, allowing for information, feel-

ings, and needs to be expressed, the inability to do so

negatively affects our relationships and mental health

due to loneliness and frustration (Holland, 2021;

Sheppard and Sebastian, 2021). In this context, aug-

mentative and alternative communication (AAC) so-

lutions can provide support to individuals with com-

munication difﬁculties, with technology boosting a

wide range of proposals (Elsahar et al., 2019). Exam-

ples include several touchscreen applications based

on pictograms and solutions integrating mechanical

https://orcid.org/0000-0002-5900-0692

https://orcid.org/0000-0002-4094-7982

https://orcid.org/0000-0002-0511-554X

https://orcid.org/0000-0002-9858-8249

https://orcid.org/0000-0002-7675-1236

or imaging methods (Elsahar et al., 2019).

The ongoing AAL APH-ALARM project

aims

at providing solutions that contribute to an increased

sense of safety and independence for persons with

Aphasia (PWAs), in different scenarios and parts of

the day. In this regard, our detailed context analy-

sis identiﬁed communication difﬁculties as one of the

most important aspects to address. It also highlighted

that existing AAC solutions are often not speciﬁcally

tailored for PWAs (e.g., by being too general in con-

tent) or fail to be adequate for important scenarios,

such as when the person is in bed, a scenario where

communication is important (whether to ask for help

when feeling unwell or for a glass of water). For ex-

ample, existing solutions typically depend on a touch-

based device, which may not be immediately reach-

able or suitable for the user’s ﬁne motor capabilities

and potential level of distress; or rely on technology

that is more cumbersome to use (e.g., brain-machine

interfaces – BCI) or that can raise privacy concerns

for the users (e.g., RGB cameras).

Furthermore, among the contributions on assistive

technology, solutions to support a person lying in bed

have not yet been explored, besides some work focus-

https://www.aph-alarm-project.com

Nunes, F., Rocha, A., Valente, A., Silva, S. and Teixeira, A.

Gesture-Based Communication for People with Aphasia While in Bed.

DOI: 10.5220/0011989200003476

In Proceedings of the 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2023), pages 237-244

ISBN: 978-989-758-645-3; ISSN: 2184-4984

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

237

ing mainly on gesture recognition (Guimar

aes et al.,

2021; Guimar

aes, 2021; Santana et al., 2022; San-

tana, 2021). Supporting the bed scenario is impor-

tant even for PWAs without any major motor limita-

tion, since many are afraid to get up, during the night,

in a potentially disoriented state, with fear of falling.

In this regard, we aim to go beyond a simple “emer-

gency” button towards a richer, albeit simple, com-

munication of the PWAs’ needs and concerns.

To accomplish our goals, we adopt an iterative

user-centred design and development methodology to

ground our proposals on the needs and contexts that

might better serve PWAs. In this regard, one of the

major challenges to deal with is the wide spectra of

communication issues that can affect PWAs, as this

works as a barrier towards open discussion and in ob-

taining feedback in the design stage.

When looking into how to tackle this aspect, we

considered that the collection of contributions from

PWAs would proﬁt from an approach that was not

merely conceptual, at ﬁrst, but grounded on tangible

proposals that could work as a common ground (An-

dersen and Mosleh, 2021). Additionally, these arti-

facts supporting the discussion should be chosen care-

fully, since they should incorporate, as best as possi-

ble, the PWAs potential needs and abilities.

To this end, we started working closely with

Speech and Language Therapists (SLTs) with expe-

rience in working with PWAs, as proxies to better un-

derstand PWAs’ abilities, needs and motivations, and

iteratively reﬁne a proposal that would then support

the discussion with PWAs.

In this work, we describe how we applied this ap-

proach to target the in-bed scenario, as the ﬁrst step

of an ambitious vision of providing support in every

division of the home, while starting from a place of

primacy. We aim at developing a minimally intrusive

system that supports the communication of a PWA,

while in bed, with someone who is not present.

Considering these aims, and regarding how the

PWA would interact with the proposed solution, ges-

tures seemed a potentially viable alternative, having

the advantage of not requiring the user to reach for

a smartphone or buttons/switches next to the bed, or

physically interact with a device while lying down

and possibly in distress.

All things considered, and guided by the syner-

gies with SLTs, this proposal is characterized by: (1)

adopting arm gestures as input for the PWA interac-

tion; (2) assisting the creation of messages to be sent

by the PWA through a sequence of simple questions;

and (3) providing multimodal feedback to the PWA.

2 CONCEPT PROPOSAL AND

VALIDATION

To develop the envisioned system, a User Centered

Design (UCD) methodology was adopted. We began

by deﬁning relevant personas and scenarios for sys-

tem usage, allowing us to deepen our understanding

of the target users, including their main motivations

and needs, and consequently propose a concept for

the system. This concept was then iteratively evolved,

by being judged appropriate, reﬁned and validated in

discussions with external SLTs. The outcome of these

steps also consolidated the extraction of requirements

of the system.

Regarding the usage scenarios, we focused on

those where the PWA is lying in bed since this is

an important scenario in their daily life, due to the

fact that they usually acquire Aphasia as a result

of a stroke and tend to belong to an older demo-

graphic (Ellis et al., 2010). The possibility of having

another stroke and not being able to ask for help is

consequently a fear of PWAs. Initial feedback from

SLTs collaborating in this work showed that PWAs

are also often afraid of getting up at night due to

the fear of falling. For these reasons, PWAs are of-

ten afraid of living alone or being left alone, which

greatly reduces their independence.

Another important aspect of our proposal is the

use of gestures by the PWA as input to support com-

munication. We consider this to be a more appro-

priate option for the considered scenario (i.e., PWA

lying in bed), when compared with other methods

used in AAC solutions. For example, gestures can

be performed without the need of moving around in

bed to reach for a device and does not require phys-

ical interaction with a device (e.g., a smartphone or

tablet with touchscreen, buttons/switches). Further-

more, arm gestures can be used even by users lacking

ﬁne motor skills. Nonetheless, the suitability of ges-

ture input was also one of the main aspects discussed

in the validation with STLs.

2.1 Persona and Scenario

Regarding the target users, our work is grounded on

previous contributions by Azevedo (Azevedo, 2022)

proposing a set of Personas for different proﬁles of

PWAs and their families. From those, we selected and

adapted the ones that we deemed appropriate to rep-

resent the target users of the solution proposed in the

present contribution: one PWA having their speech

affected, but without severe comprehension difﬁcul-

ties, and having a motor impairment on one side of the

body; and the Persona’s mother, as the potential des-

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

238

tination of the messages sent by the PWA. The adap-

tation mainly targeted the Personas’ motivations, to

align them with the bed scenario, resorting to the liter-

ature and feedback from different professionals with

experience working with PWAs.

In what follows, we present a summary of the

adopted Persona with Aphasia and an illustrative sce-

nario resulting from this process.

2.1.1 Persona – Judite

Judite is a primary school teacher, aged 38, who had a

stroke leading to Broca’s Aphasia (non-ﬂuent, but re-

tains comprehension) and paralysis on the right side

of the body. She has a 11-year old daughter, In

es,

and together they live with Judite’s parents. Judite’s

biggest challenges are related with the communica-

tion with her family and being more independent.

Her main motivation is to make the situation as un-

demanding as possible and ease the burden she feels

she puts on her family.

2.1.2 Scenario – Judite Is Feeling Dizzy

Judite is resting in bed in the afternoon when she feels

dizzy when trying to get up. Her parents went to

the supermarket for groceries and her daughter is at

school. Therefore, she activates the communication

support system using the corresponding pre-deﬁned

gesture. The system asks if she needs immediate as-

sistance, to which Judite answers with the gesture de-

ﬁned for “Yes”. Given the answer, the system sends a

message to her mother saying that Judite needs imme-

diate help. After receiving this message on her smart-

phone, her mother conﬁrms she has been notiﬁed and

is on her way.

The system informs Judite that help is coming and

then asks her a few more questions to better under-

stand the context. Judite is asked if she is in pain, and

she answers the question using the gesture associated

with the meaning “No”. She is also asked if she fell,

to which Judite replies again “No”. The following

question is if she is feeling unwell, being answered

using the gesture deﬁned for “Yes”. Meanwhile, all

these answers are being given as context to her mother

through her smartphone application, which helps her

to better understand the situation.

2.2 Concept Reﬁnement with Experts

To help reﬁne our initial concept and validate the

overall approach, we wanted to bring into the discus-

sion feedback and ideas from SLTs internal and ex-

ternal to our research team. While it may be argued

that PWAs might have been involved from the start,

this possibility was dismissed due to the complexity

of the condition, which may hinder the explanation of

our proposal and attainment of feedback, particularly

without a more tangible example. Moreover, the pre-

sentation of a prototype with potential “basic” ﬂaws

regarding, for instance, how to approach the reading

and comprehension aspects, may lead to an increased

sense of frustration for the PWAs, who may admit it

is their fault and not an issue with the system. There-

fore, we wanted to have the best possible proposal

based on the SLTs’ feedback, as proxy, aiming for a

later and more proﬁtable contact with PWAs.

It is important to highlight that the main focus of

our work is assisting communication and we were in-

terested in obtaining feedback with that perspective.

For this reason, SLTs are the ideal source of feedback,

since their focus is on communication and can offer

a greater insight on the best approach and most ap-

propriate strategies for supporting communication in

PWAs’ daily life.

The assessment by STLs was based on the presen-

tation of our initial ideas in the form of a Persona and

a scenario, as well as low ﬁdelity sketches of an as-

sistant for mediating the communication between the

users of the system, which were already the result of

joint work with SLTs working with PWAs, proﬁting

from their insights. These should work as a common

ground for exchanging ideas in this multidisciplinary

setting, addressing a set of users with a sometimes

complex condition to explain and understand, an ap-

proach that was found useful in other domains (Silva

and Teixeira, 2019).

2.2.1 Method

The evaluation was performed in structured discus-

sions with the SLTs judged adequate for conceptual

reﬁnement and validation. Two different sessions

were carried out with three practicing SLTs with ex-

perience with PWAs, none of them with prior knowl-

edge about our proposal: a ﬁrst session with two

SLTs, and a second session with another SLT. Both

sessions also included the participation of an human-

computer interaction (HCI) researcher moderating the

discussion and two Computer Scientists involved in

the design and development.

After a general introduction to the proposal, the

following four main topics were discussed: (1) ges-

ture adequateness for PWAs; (2) ﬂow of communi-

cation supported by the assistant; (3) presentation of

questions and feedback to PWAs; and (4) most no-

table motives and situations distressing PWAs in the

bed scenario. These topics were intended as a high-

level structuring of the session but any of the partici-

pants were free to ask further questions, share experi-

Gesture-Based Communication for People with Aphasia While in Bed

239

ences, or raise other related points.

Concerning the gestures, the SLTs were presented

with three gestures that were simple to perform with

the hand initially resting on the mattress: knock, as

knocking on a table; clean, sliding the hand horizon-

tally; and twist, rotating the wrist as if handling a door

knob. We asked the SLTs to rate – using a 5-level

Likert-like scale, from 1 (very easy) to 5 (very dif-

ﬁcult) – the three gestures regarding their ease to be

explained to the PWA, by an SLT, as well as to be

understood, remembered, and executed by the PWA,

while lying in bed. We also asked for suggestions

about other possible gestures.

Next, the participants were introduced to the over-

all idea of the assistant and its approach based on sim-

ple questions to establish the message to be sent. To

further illustrate the idea and how the PWA’s inter-

action would ﬂow, a diagram representing the com-

munication steps as mediated by the assistant, with

examples of speciﬁc questions that might be asked in

each question set, organized into three levels of pri-

ority, was used to support the discussion. We asked

the SLTs about the overall appropriateness of the ap-

proach and if the proposed sequence of questions was

deemed feasible for a PWA to express a need.

They were also asked about the most appropriate

ways to present a speciﬁc question to the PWA, con-

sidering that three different output possibilities (text,

graphics, and speech) were available. Finally, the

SLTs were questioned about the needs, fears, and mo-

tivations expressed by PWAs for the in-bed scenario,

to elicit a discussion leading to a ﬁrst set of mean-

ingful example questions to include and their order.

The actual set of questions on the working system will

be further reﬁned by later validation steps with PWAs

and might be personalized.

2.2.2 Results

There was a consensus that remote communication al-

lows for a higher level of reassurance not only to the

PWA, but also to their caregiver and relatives, reduc-

ing their constant need of checking in with the PWA.

The main takeaway messages resulting from the dis-

cussions with the SLTs are summarized in Table 1 and

a brief account of the rationale behind them is dis-

cussed ahead.

Gestures. Overall, the knock gesture was considered

the best gesture, followed by clean, with twist ranked

as more difﬁcult (see Table 2). One additional ges-

ture that was suggested for requesting help, by one

SLT, was moving the hand back and forth, in the air,

as if asking someone to come. This was, however,

identiﬁed as more complicated to do if the person is

covered, particularly if motor limitations are present.

Table 1: Summary of notable conclusions from the concept

discussion with experts.

Gestures • Knock and clean gestures approved

• Twist gesture might be too complicated

• No more than three gestures

Assistant • Concise and short questions are essential

• Organization of questions by priority makes

sense

Output • Multimodality is a must

• Complement questions with available answer

possibilities

• Associate the answers to the gestures

Needs and

Motivations

• Providing independence and reassurance would

be the main motivation for PWAs and care-

givers/relatives

• Even a PWA with good mobility can require

help at night

• Speciﬁc needs and their priority were identiﬁed

Concerning the number of gestures, all SLTs recom-

mended using no more than three gestures, but con-

sidered that two gestures would be the ideal choice

for most users. Nonetheless, even for a reduced num-

ber of simple gestures, the SLTs highlighted the im-

portance of keep the information about what features

are available, at all times, to minimize the need for the

PWA to remember them.

Table 2: Rating (1 - Very Easy to 5 - Very Difﬁcult) at-

tributed by the experts to each gesture, regarding their ease

to be explained, understood, remembered, and executed.

The results are the average of both sessions.

Gesture Explain Understand Remember Execute

Knock 1 2.5 2 1

Clean 1 2.5 2.5 1

Twist 1.5 2.5 3 1.5

Communication Mediated by the Assistant. The

SLTs agreed with the overall idea of how the assistant

obtains the information and how the needs are priori-

tized with the most urgent situation being the easiest

to communicate. The idea of a repeated knock gesture

leading to immediate help was also praised. Regard-

ing the questions, emphasis was given to the need of

using concise and short questions, also bringing to at-

tention the need to minimize the number of questions

that are needed to obtain relevant information on a

given situation.

Multimodal Output. Regarding information presen-

tation to the PWA, the SLTs highlighted that differ-

ent ways of conveying the same message are crucial

to overcome possible comprehension difﬁculties. In

this regard, text, images, and speech synthesis should

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

240

be used simultaneously. Additionally, the SLTs sug-

gested presenting the questions available for answer-

ing along with corresponding gesture depictions (e.g.,

video, image).

Motives and Sources of Distress in Bed. The SLTs

highlighted that even a PWA with reasonable mobility

is often afraid of getting up from bed and falling —

particularly during the night, affected by sleepiness,

and when help is often less available —, making the

system a potentially good way of dealing with several

needs including: stroke; fall; getting up from bed and

getting dressed; physiological needs (e.g., going to

the bathroom); nutrition (i.e., eating, drinking); medi-

cation; problems concerning the bed (e.g., getting en-

tangled in the bed covers). Some of these needs were

identiﬁed as having a higher priority (e.g., fall) while

others as more frequent (e.g., going to the bathroom).

2.3 System Requirements

The iterative work described above allowed the reﬁne-

ment of our initial concept, leading to the deﬁnition of

requirements of the system for which a representative

set is presented in what follows. The non-functional

requirements include the following:

• Being appropriate for use in the bedroom (namely

when the PWA is lying in bed), at any time of the

day;

• Enabling two-way communication between the

PWA and other people (e.g., caregiver, family

member, friend);

• Supporting communication on the PWA’s side

based on a small set of gestures as input, and dif-

ferent redundant modes of output;

• The gestures should be easy to understand, re-

member, and execute;

• Being as less intrusive as possible for the PWA.

In terms of functional requirements, the following

were deﬁned:

• Providing a gesture input modality to the PWA;

• Recognizing the different pre-deﬁned gestures,

relying on sensors worn by the user and/or placed

in the environment, and machine learning;

• Enabling the generation of simple messages to be

sent to the other person, aided by a virtual assis-

tant;

• Providing multiple output modalities to the PWA

(e.g., speech, text, graphics), which rely on speak-

ers and a display for presentating information;

• Providing an application that allows the secondary

user (e.g., caregiver) to receive information from

the PWA and send back a message.

3 SYSTEM PROPOSAL

Supported by the work carried out in the previous

section, we propose a system that supports people

with communication difﬁculties, namely PWAs, in

the bedroom scenario, where the user is lying in bed

alone (either resting during the day or awake during

the night). The system can be described as a facili-

tator of communication between the PWA and other

people who can be in the same home (in another divi-

sion) or outside the home.

This proposal proﬁts from previous work where

methods for the use of simple arm gestures as a po-

tential form of interaction for PWAs, making use

of wearable or ambient sensors, were proposed and

tested (Guimar

aes et al., 2021; Guimar

aes, 2021; San-

tana et al., 2022; Santana, 2021).

The current proposal, described in more detail be-

low, introduces several important advances over the

mentioned work, including: (1) addition of an assis-

tant to mediate the communication; (2) multimodal

output for the PWA (addition of text and graphics

modalities); (3) adoption of a multimodal architecture

based on the AM4I framework (Almeida et al., 2019).

Furthermore, since this proposal already incorporates

the feedback obtained in the discussions with SLTs

(section 2.2), it has a greater focus on the needs of the

target primary users (PWAs).

3.1 Bedroom Setup

An overview of the proposed system’s setup for the

bedroom, depicting the required hardware, is shown

in Fig. 1. Sensors, which can be wearables or ambient

sensors installed in the bed or bedroom (e.g., radar),

enable gesture input by the PWA. We chose this type

of sensors because they are less intrusive than other

approaches (e.g., BCI) and can be used without light

(as opposed to RGB cameras).

Figure 1: Overview of the setup of the proposed system for

supporting communication in the in-bed scenario.

The gesture interactions serve the selection of the

messages that are sent to the secondary user’s smart-

phone. This last user is any person the PWA wants to

communicate with, such as a caregiver, family mem-

Gesture-Based Communication for People with Aphasia While in Bed

241

ber, or friend. For simplicity, this user will be consid-

ered as a caregiver for the remaining of this paper.

After receiving a message, the caregiver can send

a conﬁrmation or question using an application. That

information is presented to the PWA relying on a dis-

play and speakers deployed in the bedroom. The sys-

tem includes a processing unit running most of the

interaction infrastructure and software.

3.2 Overall Architecture

We designed the architecture of the overall proposal

having in mind not only the adoption of a general

principle of modularity and decoupling among the

different components, but also our vision of an as-

sistive communication service that serves the in-bed

scenario but aims to be, as already mentioned, a ﬁrst

step towards a broader coverage of the home.

In this context, we looked into existing approaches

that might serve the integration of multiple applica-

tions, devices, and interaction options for the home

environment. Among some examples, such as Mu-

dra (Hoste et al., 2011), MIODMIT (Cronel et al.,

2019), and Cue-me™ (Openstream Inc., 2015), the

AM4I architecture and framework (Almeida et al.,

2019) seemed to provide the desired characteris-

tics. This framework, implementing the W3C rec-

ommendations for multimodal interactive architec-

tures (Dahl, 2013), shows a modular and decou-

pled design and deﬁnes standard languages and meth-

ods for their communication, fostering easier mod-

iﬁcation and extension of the system in the future.

Moreover, this framework has also been explicitly

proposed to tackle interaction with smart environ-

ments (Rocha et al., 2021).

The infrastructure, the different interaction modal-

ities available to the PWA, and the other components

that enable communication support are depicted in

Fig. 2. The Interaction Manager (IM) is the core mod-

ule, part of the adopted multimodal architecture, and

is responsible for managing the exchange of messages

between the different interaction modalities and ap-

plications, ensuring their decoupling. When the IM

receives a Life Cycle Event from a modality or appli-

cation, it forwards it to the destination that is decided

based on the relevant content.

The gesture input modality relies on data provided

by the sensors to recognize the gestures carried out

by the PWA. The decision based on the recognized

gesture is then sent to the assistant (throught the IM),

which decides the next steps, eventually leading to a

message being sent to the caregiver. The latter can use

the smartphone application to send a conﬁrmation or

further questions to the PWA. Since the information

to/from the smartphone is sent through the cloud, the

caregiver can be anywhere (e.g., another division of

the same home or supermarket). All messages that

have the PWA as destination are sent, through the IM,

to three different output modalities (speech, graphics,

and text).

3.3 Assistant

The assistant provides local feedback to the PWA and

gathers information about the reason for communica-

tion before sending any message to the caregiver. This

allows the PWA to know if the gestures are being de-

tected properly. It also helps reducing the burden on

the caregiver, since the assistant is responsible for ob-

taining most of the information from the PWA.

Three main considerations inform the design of

the assistant: (1) the PWA may have different mo-

tives/priorities for communication; (2) long questions

can be difﬁcult to understand by the PWA; and (3)

Yes/No answers are easier to give. Regarding com-

munication mediation through pre-deﬁned questions

with Yes/No answers, this was mentioned by SLTs as

a common approach to communicate with a PWA.

The interaction ﬂow between the PWA and the

assistant is illustrated by the simpliﬁed diagram pre-

sented in Fig. 3. When the PWA initiates commu-

nication, the assistant collects information about the

reason, by navigating through a set of pre-deﬁned

Yes/No questions, which are answered by the PWA

using the gestures deﬁned for the Yes/No meaning.

Figure 2: Infrastructure and software components that support communication.

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

242

Figure 3: Simpliﬁed diagram representing the approach chosen for the interaction between the person with Aphasia (PWA)

and the assistant. The three colored larger blocks represent different priorities for what the PWA may need to convey. For

each priority section, a set of Yes/No questions helps gather information about the overall topic, with further details being

obtained by a set of more speciﬁc questions.

To keep memorization to a minimum, only two

gestures are required to navigate through the possibil-

ities. These gestures are knock and clean (described

in subsection 2.2.1), representing the “Yes” and “No”

answers, respectively. Knock is also used to activate

the system. Hence, a PWA in distress and repeatedly

knocking will be able to ask for help (see Fig. 3).

The questions are organized in a hierarchy accord-

ing to their priority. For each priority level — visually

represented by the differently colored larger blocks in

Fig 3 —, the questions look to identify the domain

of the help needed among different topics (e.g., “In

pain?” or “Need food?”), and further options in that

domain aim at pinpointing more speciﬁc needs (e.g.,

“Head hurts?”, “Need water?”). To maximize com-

prehensibility, the questions are limited to just a verb

and a noun whenever possible.

4 CONCLUSIONS AND FUTURE

WORK

We proposed a new system to aid communication be-

tween a person with Aphasia (PWA) and another per-

son (e.g., caregiver, family), for the in-bed scenario.

The design of this system adopted a user-centered de-

sign approach, involving speech and language thera-

pists (SLTs) with experience in working with PWAs.

To support this approach, we deﬁned and itera-

tively reﬁned a Persona and Scenario that served as

grounds for designing the ﬁrst system concept, which

was then validated by three SLTs. The obtained feed-

back encouraged the approach taken and informed

the evolution of our proposal concerning the gestures,

output modalities, and questions asked by the assis-

tant, as well as the system’s requirements deﬁnition.

The resulting system proposal includes an assis-

tant that gathers relevant data from the PWA based on

the answers to Yes/No questions. The latter are pre-

sented to the PWA using multiple redundant output

modalities (speech, text, and graphics). The PWA is

able to answer them using a gesture input modality,

which performs gesture recognition based on a model

trained using machine learning and data from sensors

worn by the user.

Based on the literature and received feedback, we

consider that our proposal provides features that al-

ready go beyond existing work and can potentially

make a difference in assisting PWAs in their commu-

nication. This stems from tackling an important sce-

nario, often disregarded or addressed with more in-

trusive technologies, and by approaching the problem

with design choices that consider the PWAs’ needs

and condition. This is reﬂected in the assistant’s

mediation, question formulation, gesture choice, and

multimodal output.

The implementation of a prototype of the system

is already under way, including the development of a

gesture recognition model, to be used in the gesture

input modality, based on wearable sensor data col-

lected from subjects while they execute the selected

gestures (knock and clean) with both arms and in dif-

ferent postures while lying in bed. This prototype is

an important step towards allowing PWAs to be in-

volved in the discussion and thus obtaining feedback

from the end users, which will be essential to develop

Gesture-Based Communication for People with Aphasia While in Bed

243

a solution that is actually usable and useful in their

daily life.

It is important to highlight that the modular ar-

chitecture of the proposed system has the advantage

of allowing changes to a given component without

affecting the other. Therefore, if improvements are

identiﬁed during validation with end users, this allows

us to quickly change one or more components before

the next iteration. For instance, if a different pool of

gestures is suggested, we can quickly react with train-

ing new gestures and reﬁning the gesture modality. It

is also worth highlighting that the concept presented

here is applicable to other contexts where gestures

also make sense (e.g., sitting in a sofa), showcasing

a strong potential for evolution to be explored in the

future.

ACKNOWLEDGMENTS

The authors wish to thank the Speech and Lan-

guage Therapists who collaborated in the concept

design and reﬁnement. This work was supported

by EU and national funds through the Portuguese

Foundation for Science and Technology (FCT),

in the context of the project AAL APH

/ALARM

(AAL/0006/2019) and funding to the research unit

IEETA (UIDB/00127/2020).

REFERENCES

Almeida, N., Teixeira, A., Silva, S., and Ketsmur, M.

(2019). The AM4I architecture and framework for

multimodal interaction and its application to smart en-

vironments. Sensors, 19(11).

Andersen, P. V. K. and Mosleh, W. S. (2021). Conﬂicts in

co-design: engaging with tangible artefacts in multi-

stakeholder collaboration. CoDesign, 17(4):473–492.

Azevedo, C. (2022). Aﬂuentia: Communication support for

ﬂuent aphasia. Master’s thesis, University of Aveiro,

Aveiro, Portugal.

Cronel, M., Dumas, B., Palanque, P., and Canny, A. (2019).

Miodmit: A generic architecture for dynamic multi-

modal interactive systems. In Bogdan, C., Kuusinen,

K., L

arusd

ottir, M. K., Palanque, P., and Winckler,

M., editors, Human-Centered Software Engineering,

pages 109–129, Cham. Springer.

Dahl, D. A. (2013). The W3C multimodal architecture and

interfaces standard. Journal on Multimodal User In-

terfaces, 7(3):171–182.

Ellis, C., Dismuke, C., and Edwards, K. K. (2010). Lon-

gitudinal trends in aphasia in the United States. Neu-

roRehabilitation, 27(4):327–333.

Elsahar, Y., Hu, S., Bouazza-Marouf, K., Kerr, D., and

Mansor, A. (2019). Augmentative and alternative

communication (AAC) advances: A review of conﬁg-

urations for individuals with a speech disability. Sen-

sors, 19(8):1911.

Flowers, H. L., Skoretz, S. A., Silver, F. L., Rochon,

E., Fang, J., Flamand-Roze, C., and Martino, R.

(2016). Poststroke aphasia frequency, recovery, and

outcomes: A systematic review and meta-analysis.

Archives of Physical Medicine and Rehabilitation,

97(12):2188–2201.e8.

Guimar

aes, A., Rocha, A. P., Santana, L., Oliveira, I. C.,

Fernandes, J. M., Silva, S., and Teixeira, A. (2021).

Enhanced communication support for aphasia using

gesture recognition: The bedroom scenario. In IEEE

International Smart Cities Conference (ISC2), pages

1–4.

Guimar

aes, A. M. (2021). Bedfeeling - sensing technolo-

gies for assistive communication in bed scenarios.

Master’s thesis, University of Aveiro, Aveiro, Portu-

gal.

Holland, A. L. (2021). The value of “communication

strategies” in the treatment of aphasia. Aphasiology,

35(7):984–994.

Hoste, L., Dumas, B., and Signer, B. (2011). Mudra: A

uniﬁed multimodal interaction framework. In Interna-

tional Conference on Multimodal Interaction (ICMI),

page 97–104, New York, NY, USA. ACM.

Openstream Inc. (2015). Cue-me. http://www.openstream.

com/cueme. Accessed: 2022-09-20.

Rocha, A. P., Ketsmur, M., Almeida, N., and Teixeira, A.

(2021). An accessible smart home based on integrated

multimodal interaction. Sensors, 21(16).

Santana, L., Rocha, A. P., Guimar

aes, A., Oliveira, I. C.,

Fernandes, J. M., Silva, S., and Teixeira, A. (2022).

Radar-based gesture recognition towards supporting

communication in aphasia: The bedroom scenario.

In Hara, T. and Yamaguchi, H., editors, Mobile and

Ubiquitous Systems: Computing, Networking and

Services, pages 500–506, Cham. Springer.

Santana, L. F. d. V. (2021). Exploring radar sensing for ges-

ture recognition. Master’s thesis, University of Aveiro,

Aveiro, Portugal.

Sheppard, S. M. and Sebastian, R. (2021). Diagnosing and

managing post-stroke aphasia. Expert Review of Neu-

rotherapeutics, 21(2):221–234.

Silva, S. and Teixeira, A. (2019). Design and development

for individuals with asd: fostering multidisciplinary

approaches through personas. Journal of Autism and

Developmental Disorders, 49(5):2156–2172.

ICT4AWE 2023 - 9th International Conference on Information and Communication Technologies for Ageing Well and e-Health

244