Testing Usability in an ICT Solution for Care Cooperatives:

A Case Study

Thomas Würtz Jensen and Bodil Sørensen

VIA University College, Hedeager 2, Aarhus N, Denmark

Keywords: Usability, Ambient Assisted Living, Emotional Wheel Score, Healthcare, ICT Management System.

Abstract: This paper describes a usability test design in the development of an ICT management system in an EU project

under the Ambient Assisted Living (AAL) programme. The population of Europe is aging, and the elderly

population is increasing. This demographic change will create an increasing need for social and healthcare

services. The purpose in developing the ICT management system is to meet this challenge by creating an

effective ICT management system for elderly people in need of assistance. Results from the usability test give

the developers insight into features to be changed to create a user-friendly software. The methods used in this

study were standard usability tests and emotional wheel scores. Representative end users were given a number

of tasks while conducting thinking aloud tests and emotional wheel scores followed by qualitative interviews.

The test results were categorised in terms of effectiveness, efficiency and satisfaction. In general, the end

users’ response were positive. They suggested improvements and identified a number of challenges mainly

related to effectiveness. Emotions ranged from happiness and joy, especially after successfully completing a

task, to anger and frustration when obstacles occurred.

1 INTRODUCTION

The population of Europe is aging, and the elderly

population is increasing compared to the rest of the

population (Hametner and Adelman, 2012). In the

near future, these demographic changes will cause

serious challenges in social and healthcare systems

because this group of people has a higher need for

services. Therefore, new solutions must be developed

to address the increasing population.

This article describes processes in an EU Ambient

Assistant Living (AAL) project carrying the title

iCareCoops. The purpose of the iCareCoops project

was to accommodate the demographic challenges by

developing an effective ICT management system for

elderly people in need of assistance. The study

intends to develop a model to organise elderly care in

an efficient way and thereby support elderly care and

services.

The project supports management of care

cooperatives through a web based ICT management

system that supports interaction and fosters

prosperous collaboration among care providers, care

receivers and managers of care cooperatives.

The requirements collection for the management

system were that it could contain and handle all care

cooperatives in Europe. The system support the

managers tasks when it comes to organising care and

social services. The management system should

coordinate services between care providers and care

receivers. Care receivers should be able to order a

service using the system. Care providers should be

able to accept the job using the system and register

when the service is done. The system must contain a

billing module handling the payment of services.

Nine partners from eight European countries

collaborated in the development of the ICT

management system. They contributed to various

areas of responsibility. SIVECO ROMANIA SA

(Romania) and SYNYO GmbH (Austria) participated

in the development of the technical parts of the web

solution and iDeal Development (Denmark)

developed the mobile app for Android. VIA

University College (Denmark) developed a test

design exploring participants’ usability experiences.

SYNYO GmbH (Austria), ZHAW - University of

Applied Sciences Winterthur (Switzerland) and

ZDUS - Zveza društev upokojencev Slovenije

(Slovenia) conducted the tests. Senior Citizen

Cooperative Riedlingen (Germany) participated in

tests of the management system. Cooperatives Europe

(Belgium) provided contact to cooperatives.

Jensen, T. and Sørensen, B.

Testing Usability in an ICT Solution for Care Cooperatives: A Case Study.

DOI: 10.5220/0007773803190324

In Proceedings of the 5th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE 2019), pages 319-324

ISBN: 978-989-758-368-1

319

Usability testing is a form of quality assurance. In

this article, it is associated with users’ experiences

when testing the iCareCoops product. Results from

the usability tests provide developers insight into

features to change to create a useful solution. It is

essential to conduct usability tests to discover

problems and make an ICT product as useful as

possible (Gregersen and Wisler-Poulsen, 2013).

Eight attributes are associated with usability:

efficiency, effectiveness, learnability, accessibility,

satisfaction, navigability, content and interface

design (Aziz et al., 2018). However,

according to ISO

9241-110 former ISO 9241-11, three of these

usability test categories are essentially: effectiveness,

efficiency and satisfaction (Aziz et al., 2018; Aziz

and Kamaludin, 2018). Therefore, these three test

categories are used in this study. According to

Gregersen and Wisler-Poulsen (2013) the definition

of effectiveness and functionality is identical. Thus,

these terms are used interchangeably in this paper.

When testing effectiveness, the aim is to identify

if and to what extend the goals or tasks are attained.

Also, collecting suggestions for improvements is

important. Furthermore, it is important to identity

conformity with expectations as a part of

effectiveness. The aim when testing efficiency is to

identify how easily users can navigate the solution

and solve certain tasks.

The aim of this article is to contribute a method of

usability testing and present the results of this

method’s use when testing the usability of

iCareCoops products.

2 METHOD

This study’s usability test design consists of expert

tests and three other data collection methods with

selected end users consisting of thinking aloud tests,

systematic qualitative interviews and emotional

wheel scores.

Expert tests are used to explore technical aspects

and, for example, identify navigational dead ends and

other bugs. The purpose of end user tests is to

investigate how end users experience usability.

All tests were performed in the language of the

countries in question except for those carried out by

the ICT experts. These tests were in English.

Prior to developing the Prototype I a paper

prototype test was conducted (Sefelin et al., 2003).

The only inclusion criteria for care providers was

that they work at a cooperative taking care of care

receivers.

Figure 1: Plan of iCareCoops usability test design showing

which user groups testing different prototypes.

Inclusion criteria for care receivers required that

that they be more than 65 years old and in need of

some kind of health care or service. The managers,

care providers and care receivers solved tasks while

thinking aloud. After the test, managers were

interviewed to collect feedback and ideas to be

incorporated into further prototypes.

It was expected that the interviews with managers

would contribute sufficient information on usability

issues because of their preconditions. Therefore, no

interviews were conducted with care providers and

care receivers, as four to five users will expose 80%

of user interface (UI) flaws (Rubin and Chisnell,

2008). Each prototype test involves representative

test individuals from the user cohorts (Nielsen and

Landauer, 1993).

The three prototypes were tested in different

setups (see Figure 1). Prototype I was tested using an

expert test. Prototype II was tested using a thinking

aloud test in combination with an emotional wheel

score and a qualitative interview only with managers.

Prototype III was tested using an expert test followed

by a thinking aloud test and emotional wheel score

with all three end user groups.

Experts participating in the expert tests consisted

of a class of Danish ICT engineering students. All

findings reported were incorporated when developing

Prototype II.

Prototype I

Expert test

Prototype II

Managers

Prototype III

Expert test

Prototype III

Care providers

Prototype III

Managers

Prototype III

Care receivers

ICT4AWE 2019 - 5th International Conference on Information and Communication Technologies for Ageing Well and e-Health

320

Table 1: Examples of tasks for users.

The Prototype II usability test was carried out with

ten cooperative managers from two countries, five

from each country. In Prototype III, twelve managers

tested the management system. Five non-professional

care providers, two nurses, one occupational therapist

and one physiotherapist evaluated the system as care

providers. Twelve older adults in need of care

evaluated the system as care receivers. They explored

usability by performing relevant tasks in an artificial

working environment see table 1.

Tests of managers consisted of thinking aloud

tests and emotional wheel scores during task solving,

followed by systematic qualitative interviews after

performing tasks. The tasks were created to be

realistic and were mainly administrative in nature.

Prior to testing, a brief introduction was given

containing information about the purpose of the test,

interviews and testing procedure. All participants

signed informed consent forms.

Expert Test

The expert tests were conducted twice, first on

Prototype I and again on Prototype III. For both tests,

a class of 16 Danish ICT engineering students tested

the web solution and reported all technical findings,

such as navigational dead ends and errors.

Thinking Aloud Test

The thinking aloud test provides a list of identified

usability problems for performing basic tasks

(Gregersen and Wisler-Poulsen, 2013). Thinking

aloud tests are frequently used to test websites. In this

study, it was used to test the usability of the

iCareCoops management system. The test users

performed realistic tasks while thinking aloud.

The thinking aloud test requires users to verbalize

thoughts while working with the system to provide a

better understanding of their interpretations or

potential misconceptions of the interface (Nielsen,

1993). The thinking aloud test was recorded. This

provided analysers the opportunity to use the

recordings while analysing usability and emotions in

connection with tasks.

The thinking aloud test was analysed in an

inductive manner. Questions, such as ‘What obstacles

prevent users form completing tasks?’, ‘Which

interface elements are problematic or helpful?’ and

‘How easily do users solve the tasks?’ were answered

to explore the three usability categories: efficiency,

effectiveness and satisfaction.

The interviewer’s task was to ask questions if the

test person stopped thinking aloud for example, by

asking, ‘What are you thinking?’

If tasks could not be solved, the user was guided

through the problems to be able to continue.

The findings from Prototype II testing were

incorporated into Prototype III.

Figure 2: Modified Emotional Wheel Score.

The thinking aloud test was repeated for Prototype

III with managers, care providers and care receivers.

Emotions evoked during the thinking aloud test

were analysed by associating them with the task

where they occurred (see Figure 2).

Manager

Invite and remove members

Set service provider role to edit

Edit content in a cooperative profile

Add a cooperative service

Care provider

Request joining a cooperative

Add a new service to the list

Place a request

Answer a messa

Care receiver

Join a Cooperative.

Reply to a message.

Order a care service and schedule i

Testing Usability in an ICT Solution for Care Cooperatives: A Case Study

321

For all tests, a set of qualitative data was derived

from the measured usability criteria during the

session. Results from each iteration were used for

further development of the iCareCoops solution.

Satisfaction and Other Emotions

The emotion scoring method was inspired by the

emotional wheel (Fontaine et al., 2013). The

emotional wheel was used during the task solving

process to identify personal experiences in the

satisfaction category (see Figure 2).

Satisfaction is defined as how it feels to use the

solution (Gregersen and Wisler-Poulsen, 2013).

After solving each task, the type and the degree of

evoked emotions were collected. Participants could

choose between the following emotions: anger,

frustration, sadness, happiness and joy. If emotions

were evoked, they were scored on a scale from 1

(lowest degree) to 5 (highest degree). The emotional

wheel provides the opportunity to investigate and

analyse how emotions are affected when using the

iCareCoops solution. If anger, frustration or sadness

during a given task dominate the scores, then the

design of this part of the iCareCoops solution should

be reconsidered (Fontaine et al., 2013).

The emotions happiness and joy indicate

satisfaction and that this part of the product has a high

degree of usability.

Qualitative Interview

Managers tested Prototype II to explore further details

about the usability of the iCareCoops solution. An

individual motivational qualitative interview was

performed immediately after the manager solved the

tasks.

The interview was based on an interview guide.

The questions in the interview guide seek to uncover

usability criteria, such as effectiveness (including

conformity with expectations), efficiency and

satisfaction. The recorded interviews were analysed

in ways inspired by the systematic text condensation

method, a qualitative analysis strategy (Malterud,

2012). The procedure involved a transcription of the

interview, a coding process to define meaningful

categories and interpretation.

Cross Analysis

A Prototype III cross analysis between all participants

from expert tests, thinking aloud tests and emotional

wheel scores was performed and categorised for three

subjects: functionality, efficiency and satisfaction.

3 RESULTS

The test design shows an iterative interactive process

in which findings are incorporated continuously and

involves a number of tests conducted over a period of

time. Therefore, the results presented are mainly from

tests associated with Prototype III.

Experts

The amount of technical errors and bugs in the expert

test decreased considerably in the second test

compared to the first. The number decreased from 64

bugs in the first test to 16 bugs in the second.

The experts mainly found browser related bugs in

the second test. What worked using one browser did

not necessarily work using another. Several bug

reports described problems with functionality, such as

change of language, lack of button functionality,

problems with registration and lack of search field

functionality.

Managers, Care Providers and

Care Receivers

The outline of the findings presented below originates

from managers, care providers and care receivers who

completed thinking aloud tests and emotional wheel

scores. Results are divided into three categories:

functionality (including conformity with

expectations), efficiency and satisfaction and other

emotions.

3.2.1 Functionality or Effectiveness

Conformity with expectations is to answer the

question ‘How useful is the iCareCoops solution in

the user’s organisation?’ In general, users were

positive about the web solution. However, several

managers expressed concerns and were taking

precautions against how the solution could be

integrated with other ICT solutions. Users stated that

the app prototypes were practical because they can

manage tasks on the go. They also suggested certain

optimisations so that the solution matches their

expectations (e.g., a billing functionality).

A number of requests regarding the conformity of

the solution concerned consistency in colour usage

and colour coding for different roles, especially the

distinction between the menu buttons “My

Cooperative” and “Cooperatives”, and button

placement. Members of all three user groups found it

confusing that similar functions did not share a

colour. Managers found it useful that members have

ICT4AWE 2019 - 5th International Conference on Information and Communication Technologies for Ageing Well and e-Health

322

various roles, but they asked for colour coding for

different roles. Some managers and care receivers

also asked for a simplification of the menu and the

elimination of unnecessary duplicates (i.e., services

and service categories). Although managers asked for

simplification in general, they also asked for

additional features.

Users faced a number of challenges. Registration

was a problem for some users. A manager and a care

provider claimed that older adults in particular would

not be able to register themselves. It was difficult to

understand the difference between login and

registration, especially for care receivers, but also for

some care providers. This became obvious because

all of them tried to register in the login section. All

but one care provider did not find the register button

on the first attempt and instead tried to register in the

and requests to join a cooperative, care providers and

care receivers reported problems with the chat format

of the e-mails. They found it frustrating. Moreover,

some care receivers were uncertain if a message had

been sent because it was not shown on the website.

Some of them also experienced trouble exiting the

receiver dropdown menu. Thus, regarding the

message functionality, some care receivers

mentioned that they would prefer to write a

conventional email instead. One person mentioned

that she would prefer to join directly on the homepage

of the chosen cooperative instead of the iCareCoops

homepage. Care providers recommended that emails

that had been replied to should be marked as such. In

general, users claimed that the solution should be

simpler and more straightforward, especially for care

providers and care receivers, and that support in the

form of a helpdesk would be useful.

3.2.2 Efficiency

The efficiency category aimed at investigating

whether certain obstacles prevented users from

completing the tasks. All users agreed that the main

functionalities were covered by the solution. They

also identified areas for improvement. Navigation

was not easy for managers in general, and some of

them experienced it as inconsistent, but once the

principle was understood, the solution was easy to

use. In particular, inconsistencies in translations and

terminology proved to be an obstacle. It is worth

mentioning that no obstacles occurred that prevented

users from completing the tasks. However, care

providers and care receivers mentioned that the

solution was only understandable for users who were

accustomed to ICT solutions.

Users were confused by the date format

(mm/dd/yyyy). Moreover, there was no information

about choosing a password – specifically, that at least

seven letters were required to create a password.

When the password was inadequate, every written

letter or number disappeared when clicking the

“REGISTER” button. This caused frustration for all

users.

With reference to interface efficiency, care

receivers had problems with navigating the system.

The facilitator had to help them several times, and

they were not able to complete several tasks

independently.

Increasing the size of the letters appeared to be

irritating because a smaller cut-out was displayed,

causing them to lose overview.

Care providers and managers had no significant

difficulties in navigating the interface.

3.2.3 Satisfaction and Other Emotions

Various emotional reactions occurred during the

thinking aloud test. The emotions joy and happiness

correspond to satisfaction. Emotions ranged from

happiness and joy, especially after successfully

completing a task or when functions were understood,

to anger and frustration when obstacles occurred.

There was a lack of feedback when requests for a

service were given. The results in relation to

satisfaction by senior users revealed the need for

more detailed and precise feedback, as requested by

both care receivers and care providers.

Care providers expressed frustration when they

did not successfully complete registration, and they

wanted precise feedback on their actions. In general,

they experienced difficulties getting an overview of

the web solution. Overall, two tasks, registering for

iCareCoops and service requests, must be taken into

account. All other tasks scored mainly positive

emotions, which correlates with a high degree of

satisfaction and, therefore, in this category, a high

degree of usability.

4 DISCUSSION

The expert test showed that technical errors and bugs

decreased considerably in the second expert test

compared to the first. This may be a result of the error

corrections that were made when developing later

prototypes.

In this study, the satisfaction category was mainly

explored during use of emotional wheel scores. Other

studies used post interviews or diaries to document

Testing Usability in an ICT Solution for Care Cooperatives: A Case Study

323

and explore satisfaction (Bastien, 2010). The purpose

of scoring emotions during the thinking aloud tests

and task solving process is that the users do not have

to remember which emotions they experienced during

the task and to what degree they experienced these

emotions. They were instead encouraged to express

them while they felt them (Fontaine et al., 2013).

A study on mobile applications showed that

degree of satisfaction is directly associated with

functionality, such as the amount of user errors and

navigational errors (Saleh et al., 2017). Their finding

conforms to this study, in which the number of

navigational dead ends and other errors resulted in

low functionality and satisfaction scores.

Other studies have explored usability aspects in a

quantitative manner using various methods. Aziz et

al. (2018) measured the role of satisfaction. They

used a survey to quantify users’ perceptions, feeling,

opinions and thoughts. The emotional wheel method

provided the opportunity to analyse and continuously

identify subjective emotions while testing the ICT

solution. Qualitative interviews primarily explore

explanations for evoked emotions.

Regarding efficiency, it was found that the

solution was easier to operate for users who were

accustomed to ICT solutions. This result is in

accordance with the result of Aiyegbusi et al. (2018),

who found that users who were used to navigating the

internet were much faster and made fewer errors

when solving tasks than users who were not familiar

with navigating the internet (Aiyegbusi et al., 2018).

5 CONCLUSION

The aim of this article is to contribute a method of

testing usability and present the results when this

method is used for testing usability of the iCareCoops

products.

The methods used in this study were paper

prototype testing, expert tests, thinking aloud tests

and the collection of emotional reactions using

emotional wheel scores. The test design was found

useful when testing the iCareCoops web solution.

All in all, experts mainly found technical related

errors and bugs. The other test results were

categorised in terms of functionality, efficiency and

satisfaction. The representative end users were

positive in general but suggested improvements and

identified a number of challenges, mainly in the

functionality category. Emotions ranged from

happiness and joy, especially after successfully

completing a task, to anger and frustration when

obstacles occurred.

Testing this kind of management systems it is

useful to combine several methods in order to get

insightful perspective of the users experience, needs

and emotions.

REFERENCES

Aiyegbusi, O.L., Kyte, D., Cockwell, P., Marshall, T.,

Dutton, M., Walmsley-Allen, N., Auti, R., Calvert, M.,

2018. Development and usability testing of an electronic

patient-reported outcome measure (ePROM) system for

patients with advanced chronic kidney disease. Comput.

Biol. Med. 101, 120–127. https://doi.org/10.1016/

j.compbiomed.2018.08.012

Aziz, N.S., Kamaludin, A., 2018. Measuring Website

Usability Construct as Second Order Construct in

Website Usability Model. Adv. Sci. Lett. 24, 7727–7731.

https://doi.org/10.1166/asl.2018.13007

Aziz, N.S., Kamaludin, A., Sulaiman, N.S., Yacob, A., 2018.

Measuring the Role of Satisfaction in Website Usability

Model. Adv. Sci. Lett. 24, 7762–7768. https://doi.org/

10.1166/asl.2018.13014

Bastien, J.M.C., 2010. Usability testing: a review of some

methodological and technical aspects of the method. Int.

J. Med. Inf. 79, e18–e23. https://doi.org/10.1016/

j.ijmedinf.2008.12.004

Fontaine, J.J.R., Scherer, K.R., Soriano, C. (Eds.), 2013.

Components of emotional meaning: a sourcebook, Series

in affective science. Oxford University Press, Oxford.

Gregersen, O., Wisler-Poulsen, I., 2013. Usability: test

methods for making usable websites. Grafisk Litteratur,

Copenhagen.

Hametner, M., Adelman, R. (Eds.), 2012. Figures for the

future: 20 years of sustainable development in Europe?;

a guide for citizens, 2012 ed. ed, Statistical books /

Eurostat. Publications Office of the European Union,

Europäische Kommission, Luxembourg.

Malterud, K., 2012. Systematic text condensation: A strategy

for qualitative analysis. Scand. J. Public Health 40, 795–

805. https://doi.org/10.1177/ 1403494812465030

Nielsen, J., 1993. Usability engineering. Morgan Kaufmann,

San Diego, Calif.

Nielsen, J., Landauer, T.K., 1993. A mathematical model of

the finding of usability problems. ACM Press, pp. 206–

213. https://doi.org/10.1145/169059.169166

Rubin, J., Chisnell, D., 2008. Handbook of Usability Testing:

Howto Plan, Design, and Conduct Effective Tests. John

Wiley & Sons, Inc., Hoboken.

Saleh, A.M., Ismail, R., Fabil, N., Norwawi, N.M., Wahid,

F.A., 2017. Measuring Usability: Importance Attributes

for Mobile Applications. Adv. Sci. Lett. 23, 4738–4741.

https://doi.org/10.1166/asl.2017.8879

Sefelin, R., Tscheligi, M., Giller, V., 2003. Paper prototyping

- what is it good for?: a comparison of paper- and

computer-based low-fidelity prototyping. ACM Press, p.

778. https://doi.org/10.1145/ 765891.765986

ICT4AWE 2019 - 5th International Conference on Information and Communication Technologies for Ageing Well and e-Health

324