5 THE EVALUATION STUDIES
The main contribution of our framework is estab-
lishing a communication platform between the health
care staff at various levels (e.g., between the staff at
HSC and the mobile response teams) and the elderly
person at home. However, in our conducted evalu-
ation studies we mainly focused on the mobile plat-
form that we developed to be used by the mobile
ERTs.
We conducted the evaluation studies at two stages.
The first evaluation study was carried out with
three usability experts using the heuristic evalua-
tion (Nielsen and Molich, 1990; Nielsen, 1994) ap-
proach. This evaluation ran on the first implemented
prototype of the developed mobile app and the goal
was to find out the possible usability flaws in the
mobile app using the ten heuristics proposed by
Nielsen (Nielsen, 1994). Based on the results of
this evaluation study by experts, we redesigned our
mobile app user interface and made the suggested
changes.
The second evaluation study was conducted with
10 participants having different background using
the controlled task-based evaluation experiment ap-
proach. They were also given closed-ended and open-
ended questionnaires at the end of experiment in or-
der to know their feedback. These participants were
mostly researchers and students from the University
of Kaiserslautern, as the goal of the second evalua-
tion study was to analyze whether the developed mo-
bile app (which will be eventually used by ERTs) is
easy to use without any prior training or expertise.
In the following subsections, we provide details
of the conducted heuristic evaluation study followed
by the user evaluation study in the controlled environ-
ment.
5.1 The Heuristic Evaluation Study
The heuristic evaluation study was done with three us-
ability researchers from the Computer Graphics and
HCI group of the University of Kaiserslautern. They
evaluated the first implemented prototype of the mo-
bile app using the ten heuristics and ranked the user
interface giving a number from 0 to 4, where 0 means
no problem at all while 4 means usability catastro-
phe (Nielsen, 1994). The goal was to find out usabil-
ity issues in the early stage of the mobile app devel-
opment in order to improve the design and UI based
on this evaluation.
A list of tasks were given to these experts con-
sisted of the main features of the developed mobile
app: i.e., managing settings, performing status up-
dates, giving response to the emergency requests, es-
tablishing Skype call with the patient, sending feed-
back to the server, observing elderly person’s house-
hold map, route planning with Google Maps, and con-
trolling the ARTOS with joystick control option.
An introduction of the mobile app was given to
these experts before they started to evaluate it. Fur-
ther, the task description from the users’ point of view
was also given to them. They were also allowed to ask
details about any UI element or task during the evalu-
ation. In order to maintain the settings, the same per-
son was given the chance to explain the mobile app
and tasks during the evaluation. A form was given
to these experts in order to record their feedback re-
garding the ten heuristics and their overall comments
regarding any particular task or UI element.
Figure 7 shows the feedback of these three experts
using the ten heuristics, where the score indicates the
average of their given ranking in all tasks. The feed-
back indicated that some changes needed to be done
for preventing errors and extending the visibility of
the system status. However, there was not any major
usability problem or usability catastrophe. The worst
average grade of 2.0 from one expert was received
in the category of help and documentation criteria.
As our mobile app is intended to be used for pro-
fessional purposes by ERTs, help and documentation
must be sufficient for users with any level of compe-
tency and should provide in-depth review of the fea-
tures. Other major issues were in the cases of fifth and
ninth heuristics (average score of 1.25 and 1.08 by all
experts, see Figure 7), which indicated insignificant
lack of clarity on how to prevent and recover from er-
rors.
These experts also gave their suggestions for im-
proving the usability of the developed mobile app,
e.g., to introduce status fields of the rescue team and
the current emergency. These suggestions were then
analyzed and taken into account while developing the
final version of the app.
5.2 The User Evaluation Study
The goal of this conducted user evaluation study was
to analyze whether the developed mobile app is easy
to use without any prior training or expertise. There-
fore, we were interested in finding out the effective-
ness, efficiency, and user satisfaction aspects of our
developed app, as it will be used by ERTs for com-
municating with the HSC and the elderly person in
the emergency situation. The following metrics were
derived from the GQM model (Basili et al., 1994).
• Metrics for effectiveness: We measure and com-
pare the percentage of corrected completed tasks.
ICT4AgeingWell2015-InternationalConferenceonInformationandCommunicationTechnologiesforAgeingWelland
e-Health
122