A Cooking Game for Cognitive Training of Older Adults Interacting

with a Humanoid Robot

Eleonora Zedda

1,2 a

, Marco Manca

and Fabio Paternò

CNR-ISTI, HIIS Laboratory, Via Giuseppe Moruzzi 1, Pisa, Italy

Department of Computer Science, University of Pisa, Largo Bruno Pontecorvo 3, Pisa, Italy

Keywords: Humanoid Robot, Cognitive Stimulation, Serious Game, Human-Robot Interaction, Older Adults.

Abstract: In this paper, we present the design and the implementation of a cooking game for older adults interacting

through a humanoid robot. We discuss the motivations and the requirements that have driven such design and

indicate how it has been implemented. The main goal is to stimulate the cognitive resources of older adults in

order to limit their decline. For this purpose, we have exploited the multimodal possibilities of the humanoid

robot and have identified two robot personalities, which are suitable to improve users' engagement, and thus

their potential participation in cognitive training programmes.

1 INTRODUCTION

By 2050, the number of individuals over the age of 85

is projected to be three times more than today (World

Health Organization, 2021). In this scenario, most

older adults will need physical, social, and cognitive

assistance. Indeed, ageing has a considerable impact

on the health of older adults in terms of cognitive and

physical impairments, which influence the abilities to

complete and perform basic activities of daily living,

such as cooking, shopping, managing the home,

bathing, dressing.

Nowadays, a large proportion of dementia care is

provided by informal caregivers, usually family

members. These caregivers often experience a

negative impact on their psychological, emotional

and physical well-being due to the high workload.

(Carros, Meurer, Loffer, & Unbehaun, 2020). Given

the high health care expenditure at older ages, and

such effects on family caregivers, new technologies

to assists older adults with cognitive impairments are

urgently needed.

Non-pharmacological interventions, such as

physical training, cognitive training, social

stimulation activities have been used to mitigate the

cognitive decline by maintaining or improving

cognitive abilities, social well-being, and the quality

https://orcid.org/0000-0002-6541-5667

https://orcid.org/0000-0003-1029-9934

https://orcid.org/0000-0001-8355-6909

of life of older adults (Carros, Meurer, Loffer, &

Unbehaun, 2020), (Cruz-Sandoval, Morales-Tellez,

Sandoval, & Favela, 2020), (Kim, et al., 2015).

Nevertheless, traditional interventions require

experienced instructors who may be unavailable.

Assistive technologies can provide useful support to

address this problem. They are technologies that have

the aim to assist different types of users during their

rehabilitation. They can help older adults maintain

their independence during daily routines and can also

be an important instrument during their rehabilitation

(Nishiura, Nihei, Nakamura-Thomas, & Inoue,

2021).

In recent years, humanoid robots have increased

their similarity to human behaviour starting from the

gestures and facial expressions to the ability to

understand questions and provide answers. Thanks to

such human characteristics, the interaction between

people and robots is becoming more natural.

Some authors use the term Socially Assistive

Robot to indicate a robot that assists users through

social interaction and effective interactions to provide

assistance and obtain measurable progress in

rehabilitation and learning (Feil-Seifer & Mataric,

2005).

Previous work has investigated how robots can act

as therapy assistants for children with autism (Jain,

Zedda, E., Manca, M. and Paternò, F.

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot.

DOI: 10.5220/0010721500003060

In Proceedings of the 5th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2021), pages 271-282

ISBN: 978-989-758-538-8; ISSN: 2184-3244

271

Thiagarajan, Shi, Clabaugh, & Matarić, 2020), or as

a tutor or a teacher helping the scholars to create math

knowledge (Janssen, Van der Wal, Neerincx, &

Looije, 2011), nutrition, and healthy eating (Rosi, et

al., 2016), or as assistant or trainer for older adults

with cognitive impairments (Pino, Palestra, Trevino,

& De Carolis, 2020) (Manca, et al., 2021).

This work focuses on the design of a game for

older adults interacting with a humanoid robot. We

consider this kind of technology in this context

because it can potentially promote seniors’ cognitive,

physical, and emotional well-being and also reduce

the workload of the healthcare system (Vänni &

Salin, 2019).

A humanoid robot is a system that can employ

different interaction strategies, such as verbal and

non-verbal communication, the use of facial

expressions, communicative gestures, and sensors.

These capabilities are essential to creating social and

emotional interaction with the users to increase their

acceptability and users’ engagement, which may

increase the possibility to reach the goal of the

assistance in less time and with better results (Carros,

Meurer, Loffer, & Unbehaun, 2020) .

Robots to support and assist patients can be a

valuable tool to help them during their cognitive

training. In such context, digital cognitive training

through serious games (SG) may potentially benefit

those with cognitive impairments more than

traditional training due to enhanced motivation and

engagement (Manca, et al., 2021).

Serious games are digital applications specialised

for purposes different from pure entertainment, such

as education, and stimulating cognitive and physical

functions. In the literature, different studies show how

digital games can obtain positive results stimulating

older adults and helping them improve their cognitive

abilities with respect to traditional training (Tong &

Chignell, 2013).

Combining a humanoid robot and a serious game

can be an exciting solution to obtain measurable

progress in cognitive functions and stimulate the user

to continue the training programme.

The aim of this work is to present and discuss the

design of a serious game that exploits a humanoid

robot capabilities. We also indicate the requirements

considered in the design of the application for

supporting older adults in a cognitive training. A

serious game prototype has then been implemented

following the requirements, which can be used to

assess the benefits of the proposed solution. The

article is structured in four sections. Section 1

provides the motivations for this work. Section 2

discusses the state of the art in the area of robots in

cognitive training with older adults. Section 3

describes the approach proposed, mainly providing a

brief description of the serious game designed and

how it can be delivered designing two possible

personalities in the robot behaviour. Section 4 details

the design of the game proposed, the technologies

used for the implementation, the possible interaction

modes, a possible robot personalities implementation,

and reports on a preliminary evaluation with two

experts. Finally, we draw some conclusions and

provide indications for future work.

2 RELATED WORK

There have been several contributions dealing with

the use of robots for elderly care. Various goals have

been considered in such context, such as providing

social companionship, or physical or cognitive

assistance.

Typical tasks considered include assistance in

daily living activities (e.g. reminders to take

medicines), cognitive training (Manca, et al., 2021),

therapy facilitator for alleviating dementia-related

behavioural symptoms (Cruz-Sandoval, Morales-

Tellez, Sandoval, & Favela, 2020). A systematic

review shows how socially assistive robots could

potentially increase the well-being of older adults

and, at the same time, decrease the caregiver's

workload (Kachouie, Sedighadeli, Khosla, & Chu,

2014).

Concerning the benefits of cognitive training

using robots, some studies have found that training

with a robot can improve the cognitive function of

older adults. A study conducted at Osaka City

University compared a speaking humanoid Kabochan

Nodding Communication Robot with the same robot

but without the communication elements. In the 8-

week trials with 34 older adults with dementia, they

found an improvement in cognitive functions in the

communication robot version, particularly in

executive function, and verbal memory function

(Tanaka, et al., 2012). Another study, conducted by

Kim et al (Kim, et al., 2015), aims to understand the

differences between traditional cognitive training and

cognitive training with a humanoid robot. Eighty-five

older adults over 60 without cognitive impairment

were recruited for the 12-week trials. The authors

measured the cortical thickness, a change in which is

associated with cognitive decline, and the cognitive

functions before and after the intervention. The robot-

assisted cognitive training was organised using 17

cognitive training exercises, including five programs

for memory, two for language, two for calculation,

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

272

four for visuospatial function, and four for executive

function. The researchers found that participants who

received cognitive training showed less cortical

thinning than the control group. While the traditional

training obtained a better general cognitive score,

robot training produced meaningful structural brain

changes and increased executive function associated

with attenuation of the cortical thickness.

Another study (Pino, Palestra, Trevino, & De

Carolis, 2020) evaluated the effectiveness of human-

robot interaction to reinforce therapeutic behaviour

and treatment adherence. The 8-week trials involved

twenty-one older adults with Mild Cognitive

Impairments (MCI) and the humanoid robot Nao to

stimulate some specific memory areas: attention,

categorisation, and association as learning strategies.

The researchers selected five tasks for the training:

reading stories; questions about the story, associated

and not associated words, recall, and song-singer

match. In the study, the researchers found that

human-robot interaction can reinforce therapeutic

behaviour and adherence to treatment.

Serious games could be a solution to provide

cognitive training and engage the users to continue

the therapy. Various studies have evaluated different

strategies for alleviating the cognitive impairments in

older adults and evaluate how serious games may

obtain better results than traditional training.

Tapus et al. (Tapus & Vieru, 2013) evaluated if

the use of different cognitive games in a robot can

increase the cognitive attention of users. The

researchers sought to stimulate some brain functions:

sustained attention, selective attention, divided

attention; working memory, and psycho-mobility in

older adults with MCI. In the 8-month trials, nine

subjects over 70 with an MCI diagnosis interacted

with two humanoid robots (Nao and Bandit) and a

tangible interface. The cognitive training was a

music-based cognitive stimulation game. As a result,

the study indicated the overall effect of improving

user performance on the memory task in a music-

based cognitive game through the assistance of a

robotic system.

Carros et al. (Carros, Meurer, Loffer, &

Unbehaun, 2020) aimed to understand the potential of

robot-based assistance and identified enablers and

barriers to the potential implementation of the robot

systems with different serious games. In the 10-week

trials with six older adults without cognitive

impairments and four caregivers in a residential care

home, they developed cognitive and physical training

using a Pepper robot. The cognitive and physical

training was organised in three phases: motivation, in

which the users interacted with the robot using a

music quiz game; physical, in which Pepper

demonstrated some physical exercises and performed

and explained the exercises as a trainer; and finally a

cognitive phase. As a result of the study, the older

adults gave positive feedback for the interaction with

the robot. In the beginning, the residents were

reluctant to interact with the robot, at the end the

residents started to pet the robot as it were a young

child or a pet. Concerning the attitudes and

acceptability of the robot, the older adults provided

positive feedback. The researchers found that robot-

based assistance quickly became familiar with the

patients and improved their engagement and

satisfaction.

Regarding the acceptability of the robot by older

adults, Abdollahi et al. (Abdollahi, Mollahosseini,

Lane, & Mahoor, 2017) investigated if an intelligent,

emotive social robot could improve the quality of life

and acceptability of older people with dementia.

Ryan, the robot used in the experiment, was equipped

with cognitive games based on Montessori-based

activities. The cognitive training's goal was to

stimulate reminiscence and memory. The main tasks

during the training were based on activities related to

life histories of the patients, recollections of photo

albums, questions about the stories of the photos. The

trial duration was between 4-6 weeks and six older

adults had access to the robot 24/7 in the senior living

facility. As a result, the study found that the patients

were interested in having a robot as a companion and

accepted it.

In this paper, we present the design of a serious

game for a humanoid robot. Different multimodal

interactions with the humanoid robot are proposed in

order to provide a more engaging interaction and

obtain better results in terms of cognitive

improvements. The multimodal interactions are

exploited in different was in order to allow the robot

to show two different personalities. While in the

literature there has been some previous work that

used a robot with different personalities for

rehabilitation therapy (A.Tapus, C. Tapus, & Matarić,

2008), and various studies have used serious games

for cognitive training (Manera, et al., 2015), to the

best of our knowledge, this is the first time that a

humanoid robot playing serious games for cognitive

training with different personalities has been

proposed for stimulating cognitive resources, such as

memory.

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot

273

3 THE PROPOSED APPROACH

3.1 Requirements

Taking into account previous experiences reported in

the literature, and an interview with psychologists, we

have designed a serious game that requires the users

to perform different tasks designed to stimulate

various cognitive resources.

Often programmes for patients with cognitive

impairments organise the cognitive stimulations

involving the participants through different activities,

such as using serious games, performing social

activities, and physical activity with increasingly

complex exercises, music therapy, group stories, and

stretching exercises. Such different activities are

considered because sometimes the older adults feel

bored and without motivation during the training and

no longer continue the therapy (Kim, et al., 2015).

These problems can be attenuated by stimulating the

engagement in the user during repetitive cognitive

training sessions. For this reason, we propose the use

of the various interaction modalities, also to represent

different robot’s personalities, to improve the

engagement during the cognitive training and

motivate the user to continue the training, also

considering some encouraging previous experiences

in this respect (Manca, et al., 2021).

The semi-structured interview was performed

remotely with a neuroscience researcher and two

psychologists in January 2021. The main purpose of

the interview was to gather information about the

design of the game proposed, discuss the

multimodality interaction options, and the possibility

to develop different personalities in the robot.

Designing personalities for the robot interaction

may create an enjoyable interaction; increase

engagement during the tasks, enhance user’s

attention, and improve task performance accordingly.

For this purpose, we identified two possible

personalities to be performed in the sessions:

extravert and introvert personalities that can improve

the attention and the engagement of the users. Such

personalities can be exhibited in the user interaction

by exploiting the interaction modalities differently.

The relevant types of personalities have been

identified following the Big five Factor model (John

& Srivastava, 1999). The OCEAN paradigms is often

used to classify the five big personalities traits:

• Openness: represents the degree to which

someone is imaginative, curious.

• Conscientiousness: reflects the extent that

someone is careful, deliberative and self-

aware of their actions.

• Extraversion/introversion: the extent to which

an individual is assertive, outgoing, talkative,

and sociable.

• Agreeableness: is the extent to which someone

is cooperative and friendly.

• Neuroticism: the degree to which someone is

easily angered, not well-adjusted, insecure,

and lacks self-confidence (Smith, Nolen-

Hoeksema, Fredrickson, & Loftus, 2002).

The extraversion and the introversion

personalities were chosen for our project because

previous work indicates that they are the most

observable personalities (Lippa & Dietz, 2000).

Another aspect that emerged is that modulating

the multimodal interaction according to the

increasing difficulty level can be a useful support, and

can be exploited to stimulate different cognitive

areas. The organization of the game in different levels

of difficulty is necessary to evaluate the

improvements from the cognitive domain, and also to

stimulate the user to continue the therapy.

The following requirements have thus been

identified based on the interview and the state of art

analysis:

• The game must be organised into three

difficulty levels. According to the level

selected, different parameters (e.g. number of

questions, time to answer) and ways to exploit

the interaction modalities are provided.

• The vocal interaction should be supported as it

is the most immediate modality to interact

with the robot;

• The game must support different interaction

modalities: vocal, gestures, and touch.

• The game should provide reinforcement

feedback during the sessions.

• The sessions should last at maximum 40

minutes.

• The design of the visual interface in the robot

screen should satisfy the guidelines for design

an accessible interface for older adults.

• The robot should show two personalities:

extravert and introvert, and exploit verbal and

non-verbal cues to identify two personalities

according to the literature (Tay, Jung, & Park,

2014).

• The game should stimulate multiple cognitive

domains.

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

274

• The game should reproduce daily activities

that the older adult is able to perform.

• Animations (robot’s movements) may be

exploited to define the robot’s personalities

and may also increase the user engagement.

• It would be useful to provide an additional

application where the therapists can monitor

the state performance of the users during the

interactive sessions.

3.2 Game Design

Considering previous work, including a systematic

literature review (Palumbo & Paternò, 2020), and the

interview with the psychologists, we decided to

design an application connected to some daily

activity. Thus, we chose to design a cooking game

requiring users to recognise the recipe ingredient's

chronological sequence, the typology of the

ingredients, and their weight. It aims to stimulate

working memory, semantic memory, and procedural

memory. During the game, the robot shows and

vocally synthesises the ingredients for the selected

recipe. Then, it starts the quizzes, during which the

user should use visual attention and working memory

to recognise the right ingredients and select them over

other options. Additionally, the game stimulates

working memory by which the user must remember

the correct sequence for the dish preparation. In this

game, the type of interaction allowed is vocal,

graphical and touching the robot sensors. The game is

organised into three difficulty levels. For each level,

various parameters, such as speed of the sequence of

the exercises, the number of elements, the time

available to complete the task, the volume and time

available for the task are modulated.

We decided to use recipes with well-known

ingredients. In this way, we decrease the differences

between people who are expert to cook and those who

are not, and we obtain a good baseline for stimulating

and evaluating memory improvements. The three

recipes chosen are chicken curry, beef chili, and

brownies.

Hence, the designed application starts with an

initial introduction in which the robot asks the user's

name and starts introducing the game instruction, and

the recipe description. During the recipe instruction,

the robot emphasises the sequential order, the

typology of the ingredients and their weight.

After this part, the application starts with a first

question that regards the ingredient's sequence. The

user has 30 seconds to recognise the first ingredients

among the four proposed. For each question, the user

has three possibilities to answer it.

The application is organised into three different

difficulty levels: easy, medium, hard. At each

difficulty corresponds different configurations of

some parameters: the number of ingredients in the

recipe, the reaction time, the question type (questions

about the chronological sequence, the weight of each

ingredient and the specific type of ingredients used in

the recipes, e.g. the use of dark chocolate or the red

onion upon milk chocolate and white onion) and the

total number of questions.

The following table 1 summarises the different

configurations considered for the levels in the

prototype.

Table 1: Difficulty level's parameters.

Level Easy

N° recipe ingredients 4

Reaction Time 30 seconds

Typology of Question Chronological Sequence

Weight

Total number of question 8

Level Medium

N° recipe ingredients 6

Reaction Time 20 seconds

Typology of Question Chronological Sequence

Weight

Ingredients typology

Total number of question 16

Level Hard

N° recipe ingredients 8

Reaction Time 16 seconds

Typology of Question Chronological Sequence

Weight

Ingredients typology

Total number of question 20

The questions are associated with the levels as

described in Table 2.

S represents questions regarding the

chronological sequences, W stands for the questions

regarding the weight of the ingredients, T are

questions regarding the type of the ingredients (e.g.

chocolate type: milk or dark).

Table 2: Questions Sequences.

Level Easy

S W S

Level Medium

S W T

Level Hard

S W T

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot

275

According to the increment of the difficulty level

another type of interaction is provided: the touch on

the hand's robot. The touch interaction with the

robot’s sensors is enabled when the difficulty level is

increased because it requires more coordination and

attention. In particular, the touch interaction with the

robot hands is provided for the medium and difficult

levels, and is associated only with the question

concerning the type of ingredients. For example,

when the robot asks "Which is the type of onion, red

or yellow?". The user should touch the right robot

hand to answer yellow or the left robot hand to answer

red. In order to indicate the association between the

possible answer and the correspondent robot’s hand,

two stickers with the corresponding colours have

been applied on the robot’s wrist. Thus, at the binary

question, the screen shows two buttons with the two

different colours associated with each option.

Additionally, the game provides different

feedback, depending on whether the user answers

correctly or not. The feedback is modulated using

different cues: verbal cues, animations and gesture

and providing visual feedback. Feedback is provided

to stimulate and encourage the users to continue the

session, and help them to memorise the right answer.

Nonverbal feedback as animations are important

elements that can improve user’s engagement during

the interaction.

3.3 The Humanoid Robot

The humanoid robot used in this work is the Pepper

developed by Softbank's Robotics. Pepper is a 1.2-m-

tall wheeled humanoid robot, with 17 joints for

expressive body language, with three omnidirectional

wheels to move around.

Pepper has multimodal interfaces for interaction:

touchscreen, speech, tactile head, hands, bumper,

LEDs and 20 degrees of freedom for motion in the

whole body. The robot is equipped with a

LG CNS screen of 10.1 inches with a resolution of

1280x800 for supporting touch interaction.

Pepper is equipped with motors that allow it to

move the head, arms and back, six laser sensors and

two sonars, which allow it to estimate the distance to

obstacles in its environment.

Pepper can detect the provenance of sounds and

voices and turns its face in the direction of those who

are talking thanks to four directional microphones on

its head. The robot speaks or reproduces sounds

thanks to two speakers in the ears. The robot is

equipped with 4 microphones on the head. Regarding

the vision pepper is equipped with two identical 2D

cameras, a 3D camera located and a stereo camera in

the forehead. For connectivity is equipped with

Ethernet and Wi-Fi.

4 THE PROPOSED GAME

4.1 The Development Environment

For the development of the robot, Softbank robotics

provides a library called QiSDK and an android

studio plugin called Pepper SDK.

Pepper SDK, an additional plugin for Android

studio, increments the additional features provided by

Android studio. In particular, it provides a feature to

generate and modulate the gesture and the robot

joint's movement called Animation editor. The

animation editor (Figure 1) allows the user to create

and edit the animation timeline. In particular, we have

created the animations modulating different joints of

the robot: the head yaw, the head pitch, and for the

left and right arms respectively the pitch, roll, yaw

and hands.

Figure 1: Animation Editor.

Additional features provided by the plugin are related

to the robot's connection and emulation. The robot

connection allows the developer to connect to the

robot and the tablet installed on its chest. The robot

emulator allows the developer to simulate the robot in

3D and its tablet, and additionally, grants the

interaction with the robot using the Robot Viewer.

Robot Viewer is composed of a robot view interface,

motion view, dialogue view and log view. The robot

view displays the real or virtual robot in 3D. Motion

view grants the user to control the robot's monuments.

The dialogue view allows the developer to control the

dialogue outputs and enter simulated or real dialogue

input. Log view provides the log occurring during the

process.

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

276

For supporting the dialogue, a chat editor is

provided. The Chat Editor is an extension of the

Android Studio's text editor. Its main purpose is to

help the user to write chat topic files, which are files

that contain different rules, that help the robot and the

user to develop interactive vocal dialogues (Softbank

Robotics, 2021).

Figure 2 reports an example of a chat topic file

used during the introduction phase for the cooking

game.

Figure 2: Example chat topic file.

The concept name collects all the possible users

allowed to access the game. After the user says one of

the possible names in the concept, Pepper replies

saying "Nice to see you $name" and records the user's

name in the system.

A topic is an object containing rules written using

the QiChat syntax, which is a language used to write

dialog topic files. The rules associate the human input

with a relevant robot answer. In the example, in

Figure 2 a topic is defined with the chatbot name.

Then a concept is defined. A concept is a list of words

or phrases that are linked with that specific concept.

In Figure 2, the list of words allowed are the names

of the users who can have access to the game. Then

the user rule (u:) is defined, a user rule has the effect

of making the robot say or do the answer specified in

the line. In this case, if the human inputs (e.g. the user

says “Eleonora”) matches with a word containing in

the concept, the robot replies saying the answer

defined in the user rule (“Nice to see you, Eleonora”).

In the user rule, a function to execute is called, which

is an external method that is used to pass parameters

or constants to the application.

4.2 Game Interaction and

Implementation

The game is organised into three main states:

introduction, play, results.

The starting state is the introduction. At this stage,

it is possible to choose language, difficulty level,

instructions. At first, Pepper greets the user and opens

both arms over its head two times to simulate an

invitation to interact with it. Then, the user vocally

selects the preferred language between English or

Italian. In this phase Pepper simulates some

autonomous movements in order to show a social

respond. After the language selection, Pepper asks the

user's name and greets the user again. In this way, it

demonstrates its emotional nearness to the user. In

this phase of greeting, Pepper says to the user: “Hi!

My name is Pepper and I’ll help you during the

following game!”.

Simultaneously Pepper simulates a greeting with

its right arm. After that, the caregiver can vocally

select the difficulty level. Before the level selection,

Pepper provides the game instructions. In particular,

Pepper says "I’ll show you a recipe and you'll have to

answer questions about the ingredients, their weight

and type. You'll have 30, 20 or 15 seconds to answer

according to the level of difficulty chosen”, and

simultaneously it performs different gestures:

twisting its torso, slightly opening its arms and hands,

and placing its right hand on its hip and showing the

open palm to the user. This animation is performed to

simulate human behaviour and explain the

instructions also moving its arms. The animation also

aims to stimulate the user to focus on the game rules.

Then, the robot explains the steps to complete the

recipes, providing information about the ingredients,

their type and weight.

In addition, while explaining the cooking

instructions, the robot expresses more emphasis on

the elements that will be asked in the play state. To

emphasise the ingredients more, we use the Speech

Synthesis Markup Language (SSML). Its tags allow

the programmers to customise the vocal parameters,

such as voice, pitch, pauses, modify prosodic

boundary and add prominence level (W3C, 2021).

While rendering the recipe instructions, we used a

combination of different SSML tags has been used, in

particular, to emphasise the ingredients, their weight

and type (using the tag \emph=2\ and \style=joyful\).

The use of SSML provides the opportunity to

highlight and give more emphasis and receive

attention on the vocal elements pronounced by the

robot that the user should remember.

Figure 3: Introduction sub-states.

The play state is divided into two sub-states:

questions and answers. In the question state, the robot

displays and vocally renders the three types of

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot

277

questions. The upper right hand of the screen displays

a timer showing the time remaining to answer. After

the question has been presented, the countdown starts,

and the possible answers are enabled. According to

the difficulty level, four or more possible answers are

proposed in the form of buttons showing the image of

the ingredient or its weight or type. During the

question time Pepper twists its torso and gently opens

its hands. The user has three possibilities to answer

the question.

Figure 4: Play state 4a) question about sequence 4b)

question about weight 4c) question about typology.

The answer state provides various reinforcement

feedback to the user. The user interface changes

according to the user’s answers. If the answer given

is wrong, the UI is highlighted with a red band and

displays a thumbs down icon (see Figure 5).

Additionally, reinforcing feedback is provided by the

robot in the form of vocal feedback.

Wrong answer, try again!

Figure 5: Visual and Vocal Feedback for a wrong answer

and no answer given.

If the user does not provide the right answer after

three possibilities, visual and vocal feedback is

provided such as in Figure 5 right. Moreover, non-

verbal feedback is combined with the verbal and

visual feedback. In particular, in case of wrong

answer, at the same time Pepper raises its right arm

towards its face and simulates a negation by shaking

its head to the left and right twice.

Figure 6: Animation for wrong answer.

Otherwise, if the answer given is correct, positive

reinforcement feedback is provided such as “Really

good job Eleonora! The answer is right! You are

doing a great job!” and a thumb’s up icon is showed

on the screen over a green band (see Figure 8).

Moreover, a nonverbal feedback is provided. The

robot combines different gestures: at first it raises its

arms to shoulder height, moves the forearm up and

down, and nods its head twice.

Figure 7: Animation for positive answer.

Really good job Eleonora! You are doing

a great job!

Figure 8: Visual and vocal feedback for right answer.

Additionally, the humanoid robot provides different

feedback during the session using different

modalities, such as head, arms, torso movements. If

the user provides the right answer, Pepper performs

one of the following three animations:

• Pepper raises the right arm and moves its elbow

up and down, and simultaneously it nods its head

twice (see Figure 7).

•

It raises both arms over the head twice and

expresses a sound of joy.

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

278

• It nods its head three times, and simultaneously

raises the right elbow to its shoulder twice.

Otherwise, if the user provides a wrong answer or a

timeout occurs, the robot performs one of the

following animations:

• Pepper shakes its head twice to the right and left,

and then raises its hands to its face (see Figure 6).

•

The robot raises its right arm at the shoulder and

shakes its head.

•

Pepper moves the arms up and down, and then

shakes its arms.

Finally, the result state is divided into final

feedback and data results. In the final feedback state

different visual, vocal and animation feedbacks are

provided to the user.

According to the number of correct questions,

different feedbacks are vocally synthesised. If the

user provides less than four correct answers Pepper

provides an encouragement feedback: “Good Job!

See you next time to continue to improve!”.

If the user gives between four and six correct

answers, the following feedback is provided:

“Congratulation! You answer at the majority of the

questions! At next time!”.

If the user provides more than six correct answers,

the following feedback is provided: “Excellent! You

did a great job!”.

These vocal and visual feedbacks are provided to

stimulate the user to continue the therapy and feel

gratified by the interaction with the humanoid robot.

In conclusion, a table displays a summary of the user

performance during the game session (see Figure 9).

Figure 9 shows, in particular, the number of correct

answers within the three attempts, the number of

correct answers on the first attempt, the number of

incorrect answers, number of timeouts, number of

correct answers for each question type. Such data are

helpful to understand if there is an improvement in

the memory domain.

Figure 9: Example of session game results.

4.3 How the Robot Can Show

Personalities

In addition, as introduced in the requirements section,

during the users' interaction, it can be useful that the

robot shows two different types of personality traits:

introvert and extravert personality. In this section we

describe how we plan to implement them in the game.

These personalities can be expressed by various

cues. In particular, according with the literature we

have identified different parameters to use in order to

allow the robot to show these personalities (Tay,

Jung, & Park, 2014).

Table 3 gives a brief description of the non-verbal

and verbal parameters identified for the personalities

modulation. The non-verbal parameters identified

are: the robot’s intonation with three values (neutral,

joyful, and didactic); the variation of the speech rate,

pitch, rhythm and volume; and a different set of

feedbacks. The verbal cues identified are: a

customized set of animations that can simulate

different emotions, and combine complex gestures

with different angles and speed; modulation of the

movement’s speed; association of movements with

Table 3: Personality parameters.

Extravert Introvert

Verbal

Intonation Joyful Neutral

Pitch variation pitch set at 150 [50-

200]

pitch set as

default 100

Volume rate set 90 % maximum set 70%

maximum

Speak rate 170 words per

minute

150 words per

minute

Rhythms

Variation

Variation rhythm

set 2 [0-2]

Variation rhythm

set 0

Set of feedback reinforcement

feedback

neutral feedbacks

No-verbal

Gesture both arms, head,

torso movements

with big angles,

faster res

onses

one arm with

small angles,

slower response

Animations animations that

convey different

emotions

neutral

animations

Moving speed 40% faster than

introvert

movements

slower

movements

Sound movement

associated with

sounds and melody

no sounds or

melody

Autonomous

movements

autonomous

movements

few autonomous

movements

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot

279

sounds; and the possibility to perform autonomous

movements. Autonomous movements are natural

movements that give the impression that the robot is

alive (e.g. natural arms movements)

The robot can provide a different combination of

verbal and non-verbal cues for each interaction with

the users according to the personality performed. For

example, in the extravert condition, the robot asks the

questions with a joyful intonation and a high-pitched

voice, and modulating its gesture such as moving both

arms over its head. When the user provides the right

answer the extravert robot provides reinforcement

feedback and complex animations in association with

a sound that expresses joy and increasing the speed of

its animations. On the contrary, the introvert robot

during the interaction will be neutral. For example, in

the question state, the robot synthetizes the question

with a neutral intonation and a slow animation. When

the user provides the right answer the robot reacts

with a neutral intonation and a low speech rate,

providing a short feedback such as saying “Good”,

and after a few seconds, it undulates the torso left and

right twice. In summary, the extravert personality is

shown through a more joyful and active interaction

while the introvert with a more neutral and calm

interaction.

We introduce possible ways to exhibit the two

personalities because we want to investigate if these

different personalities traits that can be activated

depending on the users characteristics and

interactions, can create an enjoyable interaction;

engage more the user during the tasks; increase the

user’s attention, and consequentially if can improve

the user’s performance. The caregiver can select the

personality according to the user personality trait

identified before the cognitive training sessions.

4.4 Preliminary Evaluation

A preliminary evaluation with two experts on

cognitive training and serious games was carried out

in July 2021.

The evaluation was organized in two different

steps: first of all, the experts interacted with the robot

and the cooking game, and then a structured interview

was performed using scripted questions in order to

evaluate the game and the proposed requirements.

The experts interacted with the robot by

performing the easy and hard difficulty levels. The

two experts are both female with ages of 27 and 29.

The interview was organized into six questions

regarding general positive and negative feedback of

the interaction with the robot and the game;

information about the animations performed by the

robot; whether it can be useful in cognitive

stimulation and whether it can be improved, opinions

on the tasks performed in the cognitive game, and the

multimodal interaction, and finally a general remark

about the sessions.

The experts indicated as a positive aspect the

multimodal interaction that simplifies the interaction

and offers elements that can stimulate more attention

from the user. They found that the multimodal

interaction is the optimal interaction mode for the

target identified because the vocal interaction is

immediate and easy to use and the touch interaction

with the robot’s sensor requires, on the one hand,

more attention and, the other hand, engages and offers

the possibility to interact with the robot through

physical interaction. They stressed the effectiveness

of the multimodal interaction as the optimal solution

with elders compared to the touch interaction with a

tablet. The experts highlighted how the touch

interaction with the robot’s screen may cause some

issues. Another element highlighted was the

importance of emphasising the ingredients that will

be asked about later in the quizzes. Some negative

feedback concerned the weights shown during the

recipe instructions presentation on the screen, which

were considered too small and difficult to read. One

expert stressed as positive element that during most

of the game the vocal feedback provided by the robot

is shown also on screen, in this way the user has

double reinforcement feedbacks. As negative element

both experts indicated the instruction state. The

experts advised providing specific instructions for

each level after selecting it. Another element, is to

provide more vocal prompt about the instructions

during the interaction. For example, when the touch

sensor interaction is required, the robot could provide

a vocal prompt in which it explains that at that time

the touch interaction is required. The experts reported

the necessity to provide an additional vocal feedback

when the robot does not understand the answer given

by the user. Concerning the animations, they reported

some issues regarding one animation shown when the

user answers correctly. They indicated that the

animation shown in Figure 7 does not effectively

convey the message of the right answer, and could be

modified by simulating a victory animation. The

other animations were found immediate and coherent.

One expert stressed the importance of animations

because they create more engagement and provide a

feeling of gratification in the user. Regarding the

animations, one expert highlighted how they could

create more engagement and stimulate attention in the

user. The game seemed well organized with the

different interaction modes and the different levels.

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

280

One expert suggested a possible change in the type of

question regarding the weight of the cooking

technique required in the recipe because she believes

that remembering weight information is a difficult

task. The other expert reported the utility of the

multimodal interaction in the cognitive context

because it improves the engagement and the attention

of the user. Overall the game was found a helpful tool

because it stimulates multiple cognitive domains. The

animations are important elements and provide more

engagement and additional reinforcement feedback.

As far as the multimodal interaction is concerned,

both experts highlighted its usefulness and

immediacy, and reported that it is the optimal

interaction model for older adults with cognitive

impairments. One expert suggested the possibility to

physically interact with the robot not only with the

hands but also with the head of the robot.

4.5 Data Visualization for Caregivers

In addition, a Web application has been developed to

provide the caregivers the possibility to access the

user's performance data regarding the interaction with

games and the robot.

The robot is provided with a Wi-Fi connection

that allows storing the collected data into an external

server. The data collected are: user id, total right

answers, total wrong answer, right answer and wrong

answer for each question typology, the total number

of questions the number of timeouts performed during

the session and reaction time.

Such data are collected to analyse the evolution of

users while playing the game and then to evaluate if

there is an improvement in the cognitive level, in

particular on working memory, on the reaction time

and the attention.

The Web application shows a game report with

general information such as the number of users that

are involved in the training, the total numbers of

sessions completed, and a graph regarding the total

numbers of errors and success rate over time. At the

top of the UI, the caregiver can select the data

performance of a specific user. Figure 10 shows the

interface and provides the information about the

user's session. The top of the page reports general

information about the user: the total number of

sessions performed, the total hours taken, graphs

regarding the success and error rates. At the end of

the page a table collects the information about the

user’s performance: the number of correct answers

within the three attempts, the number of correct

answers on the first attempt, the numbers of incorrect

answers, number of timeouts, number of the correct

answers for each question type. The Web application

is an additional tool offered to the caregivers, which

may be useful to monitor the user's cognitive

improvement over time.

Figure 10: Web application showing data for the caregivers.

5 CONCLUSIONS AND FUTURE

WORK

This paper presents the design and implementation of

a cognitive serious game for older adults with

cognitive impairments proposed through multimodal

interaction with a humanoid robot. We describe the

current state of the application prototype, and also

how two robot personalities can be added to it.

Moreover, we report on some preliminary feedback

from two experts in cognitive training with serious

games.

As future work we intend to complete the

implementation of the support for the two

personalities, and test the application with older

adults with mild cognitive impairments. Furthermore,

we want to offer the caregivers the possibility of

customising some game features to provide a better

personalization for each user. We also plan to

investigate the impact of different robot personalities

on older adults, and how their use should consider the

user personality.

Finally, after validating the proposed game we plan

to derive a set of guidelines that can be generally useful

for serious games played through humanoid robots.

REFERENCES

A.Tapus, C. Tapus, C., & Matarić, M. .. (2008). User—

robot personality matching and assistive robot behavior

adaptation for post-stroke rehabilitation therapy.

Intelligent Service Robotics, 169-183.

Abdollahi, H., Mollahosseini, A., Lane, J., & Mahoor, M.

(2017). A pilot study on using an intelligent life-like

robot as a companion for elderly individuals with

dementia and depression. IEEE-RAS 17th International

Conference on Humanoid Robotics, (pp. 541-546).

A Cooking Game for Cognitive Training of Older Adults Interacting with a Humanoid Robot

281

Carros, F., Meurer, J., Loffer, D., & Unbehaun, D. (2020).

Exploring human-robot interaction with the elderly:

results from a ten-week case study in a care home.

Proceedings of the 2020 CHI Conference on Human

Factors in Computing Systems, (pp. 1-12).

Cruz-Sandoval, D., Morales-Tellez, A., Sandoval, E. B., &

Favela, J. (2020). A social robot as therapy facilitator in

interventions to deal with dementia-related behavioral

symptoms. Proceedings of the 2020 ACM/IEEE

International Conference on Human-Robot Interaction

(pp. 161-169). ACM/IEEE.

Feil-Seifer, D., & Mataric, M. J. (2005). Defining socially

assistive robotics. IEEE 9th International Conference

on Rehabilitation Robotics,, (pp. 465-468).

Jain, S., Thiagarajan, B., Shi, Z., Clabaugh, C., & Matarić,

M. J. (2020). Modeling engagement in long-term, in-

home socially assistive robot interventions for children

with autism spectrum disorders. Science Robotics,

5(39).

Janssen, J. B., Van der Wal, C. C., Neerincx, M. A., &

Looije, R. (2011). Motivating children to learn

arithmetic with an adaptive robot game. International

conference on social robotics (pp. 153-162). Springer.

John, O. P., & Srivastava, S. (1999). The Big Five Trait

taxonomy: History, measurement, andtheoretical

perspectives. Handbook of personality:Theory and

research, 2nd ed.

Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M. T.

(2014). Socially assistive robots in elderly care: a

mixed-method systematic literature review.

International Journal of Human-Computer Interaction,

30(5), 369-393.

Kim, G., Jeon, S., Kiho, I., Kwon, H., Lee, B., & Kim Y,

G. (2015). Structural brain changes after traditional and

robot-assisted multi-domain cognitive training in

community-dwelling healthy elderly. PloS one, 10(4).

Lippa, R. A., & Dietz, J. K. (2000). The relation of gender,

personality, and intelligence tojudges’ accuracy in

judging strangers’ personality from brief video

segments. Journal of Nonverbal Behavior, 24, 25–43.

Manca, M., Paternò, F., Santoro, C., Zedda, E., Braschi, C.,

R., F., & Sale, A. (2021). The impact of serious games

with humanoid robots on mild cognitive impairment

older adults. International Journal of Human-

Computer Studies, 145, 102509.

Manera, V., Petit D.P., Derreumaux, A., Orvieto, I.,

Romagnoli, A., Lyttle, G., Robert, (2015). Kitchen and

cooking, a serious game for mild cognitive impairment

and Alzheimer’s disease: a pilot study. Frontiers in

aging neuroscience.

Nishiura, Y., Nihei, M., Nakamura-Thomas, H., & Inoue,

T. (2021). Effectiveness of using assistive technology

for time orientation and memory, in older adults with or

without dementia. Disability and Rehabilitation:

Assistive Technology 16(5), 472-478.

Palumbo, V., & Paternò, F. (2020). Serious games to

cognitively stimulate older adults: a systematic

literature review. PETRA (pp. 27:1-27:10). ACM Press.

Pino, O., Palestra, G., Trevino, R., & De Carolis, D. (2020).

The humanoid robot nao as trainer in a memory

program for elderly people with mild cognitive

impairment. International Journal of Social Robotics,

12(1), 21-33.

Rosi, A., Dall’Asta, M., Brighenti, F., Del Rio, D., Volta,

E., Baroni, I., & Scazzina, F. (2016). The use of new

technologies for nutritional education in primary

schools: a pilot study. Public health, 50-55.

Smith, E., Nolen-Hoeksema, S., Fredrickson, B., & Loftus,

G. (2002). Atkinson and Hilgard’s Introduction to

Psychology. . Wadsworth .

Softbank Robotics. (2021, 07 10). Pepper SDK-Tools.

Retrieved from QiSDK: https://qisdk.softbank

robotics.com/sdk/doc/pepper-sdk/ch3_tools/tools.html

Tanaka, M., Ishii, A., Yamano, E., Ogikubo, H., Okazaki,

M., & Kamimura, K. (2012). Effect of a human-type

communication robot on cognitive function in elderly

women living alone. Medical science monitor:

international medical journal of experimental and

clinical research, 18(9).

Tapus, A., & Vieru, A. M. (2013). Robot cognitive

stimulation for the elderly. International Work-

Conference on the Interplay Between Natural and

Artificial Computation (pp. pp. 94-102). Springer.

Tay, B., Jung, Y., & Park, T. (2014). When stereotypes

meet robots: the double-edge sword of robot gender and

personality in human–robot interaction. . Computers in

Human Behavior, 38, 75-84.

Tong, T., & Chignell, M. (2013). Designing game-based

cognitive assessments for elderly adults. Proceedings of

the First International Conference on Gameful design,

research, and applications, (pp. 127-130).

Vänni, K., & Salin, S. (2019). Attitudes of Professionals

Toward the Need for Assistive and Social Robots in the

Healthcare Sector. Korn O. (eds) Social Robots:

Technological, Societal and Ethical Aspects of Human-

Robot Interaction. Human–Computer Interaction

Series. Springer, Cham.

W3C. (2021, 07 20). Speech Synthesis Markup Language

(SSML). Retrieved from https://www.w3.org/TR/

speech-synthesis/

World Health Organization. (2021, 07 12). Ageing and

health. Retrieved from World Health Organization:

https://www.who.int/news-room/fact-

sheets/detail/ageing-and-health

Humanoid 2021 - Special Session on Interaction with Humanoid Robots

282