Usability Evaluation of a Collaborative Health Information System

Lessons from a User-centred Design Process

Berglind Smaradottir

, Santiago Martinez

, Elisabeth Holen-Rabbersvik

Torunn Vatnøy

and Rune Fensli

Department of ICT , University of Agder, Jon Lilletuns vei 9, N-4879 Grimstad, Norway

Department of Psychosocial Health, University of Agder, Grimstad, Norway

Department of Health and Nursing Sciences, University of Agder, Grimstad, Norway

Keywords: Health Information System, User-centred Design, Usability, Inter-municipal Cooperation, Dementia

Assessment, Human-computer Interaction.

Abstract: In Norway, a recent health reform urged municipalities to implement new primary health care services for

their citizens. In order to optimise resources, municipalities have established inter-municipal coordination

(IMC) to collaborate across organisational borders. Information systems become a necessary tool to support

collaboration and shared access to information in an IMC. In this context, the research project eHealth-

extended Care Coordination identified a specific need for a collaborative information system for the process

of evaluation and assessment of dementia in IMC teams. This paper presents the usability evaluation of a

collaborative information system for dementia assessment built using a user-centred design approach.

Mixed methods such as observations, semi-structured interviews and a questionnaire were used for data

collection. The results showed that the new information system supported the collaborative work of the

inter-municipal dementia team with a sufficient level of satisfaction among the end-users. The prototyped

solution established the foundations for the system implemented in the Norwegian trials of the FP7 EU

project United4Health, dedicated to Point-of-Care Services.

1 INTRODUCTION

In Norway, the Coordination reform (Norwegian

Ministry of Health and Care Services, 2008-2009)

addressed the continuity of care in national health

and care services. Services that traditionally were

carried out by specialised health care were

transferred to primary health care provided by

municipalities. Small and medium size Norwegian

municipalities faced the challenge of providing

specialised services to their citizens, accomplishing

the need for structural, organisational and

technological changes. This brought to light the need

for an effective coordination and collaboration

across organisational borders.

In this context, the research project eHealth-

extended Care Coordination (Samhandling uten

grenser) 2011-2015, focused on information flow in

inter-municipal cooperation (IMC) health care

teams. In the first phase of the project, a field study

identified the need for a collaborative information

system (CIS) to improve the information flow in

IMC health care teams. In the second phase, an IMC

dementia team participated in a user-centred design

(UCD) process entailing user workshops, laboratory

evaluations and interviews for developing a

functional prototype for a CIS for dementia

assessment (Smaradottir et al., 2015b). In the third

phase, a usability evaluation of electronic dementia

assessment forms for home visits and a

videoconference solution for collaborative report

writing were performed with the participation of an

IMC dementia team (Smaradottir et al., 2014).

This paper reports from the fourth phase of the

project. The final version of the CIS was developed

and a usability evaluation was carried out together

with end-users in order to validate whether the

system accomplished acceptable levels of

effectiveness, efficiency and satisfaction. In

addition, reflections from the UCD process that

involved the IMC dementia team are presented.

The research questions (RQs) of this study were:

RQ1: How can an information system be evaluated

taking into account the needs and requirements of

306

Smaradottir, B., Martinez, S., Holen-Rabbersvik, E., Vatnøy, T. and Fensli, R.

Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process.

DOI: 10.5220/0005703603060313

In Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2016) - Volume 5: HEALTHINF, pages 306-313

ISBN: 978-989-758-170-0

the end-users for collaborative access and

information sharing by an inter-municipal team of

dementia assessment?

RQ2: What lessons and methodological

procedures from this study are transferable and

applicable to development of technological solutions

for other clinical assessment workflows?

2 RESEARCH BACKGROUND

Dementia is a clinical syndrome with deterioration

of mental abilities and cognitive skills (Gustafson,

1996). For assessment of the cognitive aspects of

dementia, a widely used method is the cognitive

mental status examination, the Mini-Mental State

(MMS), (Folstein et al., 1975). The MMS is a set of

questions whose scored answers result in a category

of cognitive mental status.

A dementia plan was implemented in Norway in

2007 (Norwegian Ministry of Health and Care

Services, 2007), aiming at improving capacity,

competence and quality in dementia care and

enhancing the need for qualified competence in

primary care. However, due to the small or medium

size of many Norwegian municipalities, specialised

IMC dementia teams have been established

(Directorate of Health, 2011) to collaboratively carry

out the assessment of people with dementia in

neighbour municipalities.

Recently, a Delphi study with experts in

coordination and IMC in health services reached

consensus about the challenges concerning

electronic communication. Specifically, the lack of

tools impeded the collaboration of IMCs (Holen-

Rabbersvik et al., 2013). Therefore, IMC dementia

teams face challenges generated by their nature of

operation, such as limited information flow across

the municipalities and interoperability problems

between different information systems (IS).The aim

of developing a CIS for IMC dementia teams was to

provide a platform that supported the information

flow and collaborative work across municipal

borders.

An effective IS requires a detailed analysis of

end-users’ needs to inform system design. In

addition, the usability of such application is crucial

for the continuous, efficient and satisfactory use of

the system. In system development, the approach of

UCD involves end-users throughout the each stage

of the development cycle

(Lazar, 2006; Gulliksen et

al., 2003; Nielsen 1993). UCD considers the needs

of the end-users through field studies, evaluations

and task analysis, helping to understand context of

use and workflow, which are key elements for the

construction of an IS for a clinical workflow (Chan

et al., 2011; Goldberg et al., 2011). In addition,

usability evaluation is necessary to analyse user’s

interaction and user satisfaction with the system

(Jaspers, 2009; Kushniruk and Patel, 2004; Lazar et

al., 2010).

3 MATERIALS AND METHODS

The prototype from the earlier phases of the

eHealth-extended Care Coordination project was

further developed by an industry partner as a full

functioning version of the CIS which was

implemented within the secure Norwegian Health

Network (NHN, 2015). The evaluation of the CIS

was executed during two days in June 2015 and

entailed three steps: (1) test in usability laboratory

with end-users, (2) individual questionnaire and (3)

group interview. A mixed methods research

approach was used including observations,

interviews and a questionnaire.

3.1 Usability Evaluation

The usability evaluation was made with end-users in

a laboratory. The facilities had two rooms (test and

observation) connected through one-way mirror

(described in Gerdes et al., 2014). In the test room,

the system was accessed and used on a laptop

connected to an external screen and keyboard. In the

observation room, the evaluation was followed by

the research team in real-time through four monitors

connected to two stationary computers.

5 participants (4 female, 1 male; aged 41-57,

average 55.6 years) with the professions nurse, nurse

coordinator and social educator, took part in the

tests. They were all members of an IMC dementia

team from 4 municipalities. They reported an

average of 16.8 years of experience using clinical

systems and evaluated their computer skills as

medium.

Each test session started with a pre-test interview

with questions about background and experience

with clinical systems. A member of the research

team moderated each session. Participants were

asked about their first impression of the graphical

user interface (GUI). A concurrent Think Aloud

protocol (Jaspers, 2009; Kushniruk and Patel 2004;

Ericson and Simon, 1980) was employed. The task

list included 9 differentiated tasks to perform within

the system. After each task, the participants were

asked to score the task solving into five categories:

Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process

307

very easy, easy, medium, difficult and very difficult.

The tasks were based on the IMC dementia team

workflow description from the UCD workshops

(Smaradottir et al., 2015b). The test sessions had

duration of 39 to 62 minutes (average 47 minutes).

3.2 System Usability Scale

In order to evaluate the user satisfaction, the

participants individually answered the post-test

questionnaire System Usability Scale (SUS)

consisting of with 10 questions (Brooke, 1996).

3.3 Group Interviews

In order to complete the feedback, two post-test

semi-structured group interviews (n=2, n=3) were

conducted to qualitatively analyse the output of the

test (average duration 37 minutes). The CIS was

shown on a screen during the interviews, allowing

the participants to follow in detail the GUI and

comment on its functionality. The main findings

from the usability evaluations were also discussed.

3.4 Data Collection

Audio-visual recordings were made with two

cameras (1 fixed and 1 portable) and a screen

capture tool (in usability evaluation) merged into

one single video file using the software Wirecast

v.4.3.1. The recordings (.mov format) were imported

into QSR NVIVO 10 for transcription and a

qualitative content analysis (Lazar et al., 2010). This

study was approved by Norwegian Social Science

Data Services (NSD, 2015) with project number:

37920. All participants signed a consent form.

4 RESULTS

The results are presented following the three steps of

the evaluation.

4.1 Usability Evaluation

The test started with the screen patients’ overview

(see Figure 1) and questions about participants’ first

impression of the GUI. They generally stated that

the screen was useful to get a fair overview of

patients. Three participants positively commented on

the search function used to find a specific patient.

About the GUI, comments highlighted the

appropriate choice of colours, with the exception of

poor readability and contrast of black text over blue

Figure 1: GUI of patients’ overview.

background in patients’ overview screen. It was

pointed out that it could be difficult to read white

text sections, especially in rooms with bright light.

In addition, the insufficient font size both in text and

headings was stated recommending to adapt the GUI

to the full screen size. Suggestions included being

able to run a search writing only 3 letters and

increasing the speed of the search results.

On each individual patient’s view, comments of

the GUI (see Figure 2) confirmed the abundance of

colours, intended to visually inform about the

sections’ functionality. In this line, participants

commented: I liked the choice of colour and graphic

design. Very clear and easy to read. When you are

working on a patient, the colours can tell you where

you are. Patient’s key information was coloured as a

yellow section and placed at the top right.

4.1.1 Task Performance

All 5 participants successfully solved all the tasks,

with different degrees of help from the moderator.

 Task 1: Add a new patient to CIS

Participants had to click the ‘+’ sign to access

administrative functionalities and be able to

Figure 3). The task was unanimously scored

as easy. 2 participants had errors with the

Figure 2: Individual patient’s view.

HEALTHINF 2016 - 9th International Conference on Health Informatics

308

Figure 3: GUI of administrative functionalities.

input format while registering patient’s

birthdate, having to try few extra times.

Suggestions were made about having text

boxes with the exact format of the field to

avoid errors. Error messages would have to be

written in colour to improve readability. When

typing a post code, the city would have to

automatically appear. The labelling of the

button to register a new patient was suggested

as save instead of create as a more intuitive

description The list of patients was suggested

to be sorted either alphabetically or

chronologically; for the chronological order,

the newest patients would be placed at the top.

 Task 2: Add General Practitioner to CIS

To solve the task, the administrative

functionality of the GUI had to be accessed

and health care professional be chosen as

action for data input. All participants needed

help to solve the task, one participant needing

up to nine attempts. Task was scored as

difficult and finding administrative

functionality was tagged as problematic. 2

system errors were identified relating to

repetition of information: 1) when clicking

twice on create. In this case, patient was

stored twice with the same name without

notifying the user. 2) when typing a long

email address the phone number field became

invisible due to of lack of space.

Comments on navigation issues in the GUI:

Information input was ok, but the navigation

was difficult. The task was difficult to solve,

because the problem was navigation.

 Task 3: Add relative into CIS

To solve the task, the administrative

functionalities of the GUI had to be accessed.

Then, health care professional had to be

chosen and change the role to relative for data

input. 4 participants successfully solved the

task without help; one of them tried few times

before succeeding and another asked for help.

3 participants scored the task as easy, 1 as

medium and 1 as difficult. Participants

suggested being able to add different types of

relatives such as closest relative, friend,

guardian or other. They also suggested that it

would have been preferable to be able to make

a priority list of whom to contact in case of

multiple relatives registered. Comments

related to understanding how the roles were

interpreted in CIS: I found health care

professional but did not understand that it was

the right one, and the role had to be changed

to relative. It is difficult when I have not seen

the system before… Difficult to navigate, the

input was easy.

 Task 4: Navigate to patient’s view in CIS

To find the new patient’s view, firstly the icon

home had to be selected and then selecting

patient’s name in order to enter patient’s view.

4 participants successfully solved the task and

one needed help after two incorrect actions.

 Task 5: Add a task into the Patient’s View

It was necessary to click on ‘+’ symbol in the

section Tasks to solve the task. 3 participants

successfully solved the task, although 2

needed help: I did not see the heading Tasks…

I did not see tasks, did not understand to

watch on top. 2 participants scored the task as

easy

, 1 as medium and 1 as difficult. One word

regarding who to perform an action was

misunderstood and that led to confusion.

 Task 6: Upload a referral into CIS

Participant had to click the ‘+’ symbol in

Documents section and upload a document to

solve the task. 2 participants successfully

solved the task and 3 needed help. 2

participants evaluated the task as easy, 1 as

medium and 1 as difficult.

 Task 7: Upload a dementia assessment

report into CIS

The task was similarly solved as task 6,

adding a document and uploading it. All

participants successfully solved the task and

graded it as easy: Now I have tried this once

before.

 Task 8: Upload the clock-test into CIS

The task was solved similarly as task 6 and 7,

adding a document and uploading it. All

participants successfully solved the task and

graded it as easy: Now I start to understand

how the program is organised.

Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process

309

 Task 9: Write a journal note into the just-

registered patient’s view

Participants had to click on the ‘+’ symbol in

the Journal note section to solve the task. All

5 participants successfully solved the task and

graded it as easy.

4.2 System Usability Scale

The scores of the SUS questionnaire are presented in

Table 1. The colour visualisation scheme presented

is a modified version of MacLellan et al., 2012, and

Bangor et al., 2009. Overall, the mean of the

satisfaction ratings were on the range of Agree,

Strongly Agree or Neutral for the majority of

answers to the positively enunciated questions and in

the range of Disagree, Strongly Disagree or Neutral

for the majority of answers for the negatively

enunciated questions.

Table 1: Responses of System Usability Scale (SUS).

Question P1 P2 P3 P4 P5 M SD

Q1 3 3 4 4 3 3.4 0.5

Q2 4 2 1 3 3 2.6 1.1

Q3 3 4 4 3 3 3.4 0.5

Q4 1 2 1 3 1 1.6 0.9

Q5 4 4 5 4 3 4.0 0.7

Q6 2 2 1 3 2 2.0 0.7

Q7 5 4 4 4 4 4.2 0.4

Q8 2 5 1 2 2 2.4 1.5

Q9 3 4 3 3 3 3.2 0.4

Q10 2 4 1 3 2 2.4 1.1

Pi = participant i; M = mean; SD = Standard Deviation

Positive Response: Agree or Strongly Agree for positive

questions; Disagree or Strongly Disagree for negative

questions

Neutral: neither Agree nor Disagree

Negative Response: Agree or Strongly Agree for

negative questions; Disagree or Strongly Disagree for

positive questions

4.3 Post-test Group Interviews

The group interview results are presented in four

categories.

4.3.1 Test Scenario and Procedure

Participants defined the test experience as exciting

and similar to the earlier tests. They found that the

questions made after each task to grade the difficulty

of the task accomplishment were slightly more

difficult to answer. A participant commented on it: I

did not find everything, but still I don’t think this

was a complicated program. When you receive help

once, then you learn how to do it and it is easier next

time. If I had used longer time during each task I

would have probably found it by myself.

4.3.2 User Training

Participants suggested that having user training in

advance would have been useful and, in addition,

would help them to provide more feedback. They

commented that in their daily basis, they have user

training when new functions and updates are

implemented in the systems. One participant

commented: If I had been allowed to sit 10 minutes

by myself to explore the system, the test would have

been much easier. […] If we had been instructed in

advance about the three main elements I would have

understood the structure earlier.

4.3.3 Navigation

Participants found the data input for all tasks easy

with good visibility of the displayed information.

Participants found difficult how to access patient’s

journal from the administrative section (task 4). In

this line, one participant commented: It was not

obvious; I would not have found it without help. In

general, they suggested as few clicks as possible,

displaying some information only in request (e.g.,

email address, contact information to GP) using

icons.

4.3.4 Municipal EHR versus CIS

The IMC dementia team used a municipal electronic

health record (EHR) system in their daily work.

Even though participants were positive earlier in the

research project towards the new collaborative

information system, in this evaluation they

expressed some scepticism about the co-existence of

the new system with the ones previously used by the

team: I would find it a bit cumbersome to have two

different systems, one system for the inter-municipal

dementia team, and one for everything else. […] I

would not like to change the system we have now,

since we would have two systems to use. I don’t

think that is smart and would be more difficult to

work.

One of the reasons argued was that the initial

circumstances when the project started have changed

throughout the project period. One participant

commented: We get new tasks all the time and that

demands more from us. We need to ease the working

processes as much as possible. We should not have

too many programs to use. I am afraid that this

system will cause double work, instead of having one

HEALTHINF 2016 - 9th International Conference on Health Informatics

310

single system. Four years ago, I was much more

positive, because then, we did not have e-messages

or access to EHR systems in other municipalities.

Some of the problems we had at that time are now

solved. The implementation of e-message (1.5 years

ago) did revolutionise our daily work. The dementia

report is now sent as e-message. In addition, we are

now used to the tablet and to take a picture of the

clock test to upload into the municipality EHR,

instead of scanning as we did earlier. So there are

fewer papers involved now.

Another participant stated: Instead of implementing

a new system, I would suggest a collaborative space

in the [municipal EHR]. That would be helpful, with

collaborative access for the inter-municipal

dementia team to the patients undergoing dementia

assessment.

Another reason was that, when the project

started, the IMC dementia team was recently

established and they were inexperienced as a

collaborative team. Since then, they have had over a

100 dementia assessments. Routines have been

improved and less time is now used on each home

visit and in the report writing. In addition, the laws

regarding shared access to medical information

across health organisations have been changed

during the project period, and the nurse coordinator

now had acquired legal access to the EHR systems

in the involved municipalities (even though with

separate username and passwords for each system to

sceptical attitude towards implementation of the

CIS, it was stated: I like this new system and would

find it helpful. In [municipal EHR] there are too

many clicks and the information input is much more

complicated. Another participant commented:

Anyhow, I think this system would be useful. In

[municipal EHR], I need to search a lot for

information. I liked the visibility of the key

information.

Overall, participants positively commented the

participation in the research project: The

participation in this project has been interesting.

They received the news that the outcome of the

earlier phases of this project informed the creation of

another IS for remote monitoring of COPD patients:

Nice to hear that what we have participated in has

been used in another system, living its own life. […]

So our contribution already has come to use.

5 DISCUSSION

In this paper, the usability evaluation of a

collaborative information system for an IMC

dementia team has been presented. The aim of the IS

was to provide a platform that supported the

information flow and collaborative work across

municipalities’ borders. An effective IS requires a

detailed analysis of end-users’ needs, preferences

and suggestions to inform system design. For this

reason, a UCD process was employed involving

end-users in design and evaluation throughout the

entire development cycle.

The two research questions presented at the

beginning of the paper were answered based on the

results of this study. About the RQ1, which asked

about how to take into account user needs and

requirements in the evaluation of a new IS, the study

showed that the mixed methods approach efficiently

considered user needs in the evaluation of the

system. The approach was divided in three stages.

The first stage was the evaluation in the usability

laboratory, were participants performed a series of

tasks based on the IMC dementia team workflow

description provided by the users in earlier UCD

workshops. This test enabled users to give useful

feedback and first impressions about the GUI,

functionality and interactions with the system. The

second stage included a questionnaire (SUS) with 10

questions related to user satisfaction after task

solving. It showed that, overall, participants were

generally satisfied with the use of the system. The

third stage included post-evaluation semi-structured

group interviews that allowed participants to discuss

the main findings with the research team and

spontaneously make any suggestions. This stage

gave the opportunity to participants to make

comments and exchange impressions in a group,

rather than individually, what presented the research

team with new situations to learn from and which

were not previously considered (e.g., slight

reluctance to final implementation due to potential

integration problems with coexisting systems and

user work overload).

Several lessons were learned during the UCD

process that can be transferable for the development

of solutions for other clinical assessment workflows

(RQ2). Firstly, the creation of clinical systems

requires active and continuous involvement of the

end-users in the design and evaluation of the

solution. Secondly, the circumstances for the context

of use may change over the study’s time span. The

nature of this research was linked to a Norwegian

research project with the time duration of four years.

Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process

311

The key requirements for the system that were

gathered in a field study and several user workshops

in an early project phase changed as the project

evolved. For instance, new functionalities provided

and included in the collaborative information system

were, during the project time, also implemented in

parallel in existing systems. At the end of the

project, this resulted in a reduction of end-user

interest in using the new system because they

reported that improvements were already in place in

existing systems. In addition, due to recent law

changes, shared access across municipal borders was

now allowed improving information flow and

electronic communication. Thirdly, new system

integration with existing systems is vital to, at least,

not increase user workload. This is a logical

consequence of the previous lesson.

There were some limitations associated to this

study, such as the use of a simulated test

environment and a reduced number of end-users.

Firstly, although the laboratory setting realistically

simulated the work environment and representative

end-users carried out the tests for validation of the

system, the study was performed in a simulated

instead of real environment. This should be seen as a

first step in the validation, complemented by a test

of the system in real clinical settings through a field

trial would be recommended before final

implementation. Secondly, the reduced number of

participants in the usability evaluation might be seen

as an impediment of the applicability of the findings

in a larger scale. However, the participants

meaningfully represented the end-users of the

system and in qualitative usability studies, a small

number of participants can be sufficient for having

valid results (Nielsen, 1994; Nielsen and Landauer,

1993).

6 CONCLUSIONS

This study was framed inside the project eHealth-

extended Care Coordination, which aimed to

develop a collaborative information system to be

used in dementia assessment to improve the

information flow between the members of an inter-

municipal team. The system would ideally be the

core for IMC health care teams, potentially

adaptable for other clinical workflows. A UCD

process was employed throughout the whole

duration of the project, in which all the versions of

the system were evaluated and tested. The usability

evaluation, together with graphical assessment and

group interviews of the system, identified

refinements in order to improve the functionality and

effectiveness of the solution before implementation.

The SUS questionnaire showed a high score of user

satisfaction.

The time span of the project, to which this study

belonged, was four years. This period represents a

substantial amount of time in clinical environments,

usually associated with an increased demand for

technological solutions that quickly and easily adapt

to continuously evolving workflows, requirements

and existing systems. Therefore, when implementing

a new system, functionality should not duplicate the

one from existing systems. In addition, there is a

need of rapid development of new ICT capable of

integration with other parallel activities and systems.

These systems are typically used within

organisations facing continuous changes as in the

health care services.

The initial GUI of the CIS for dementia

assessment established the foundation for the user-

centred design and development of an information

and management system for remote telemedicine

monitoring of COPD patients at home (Smaradottir

et al., 2015a), which has been implemented in the

FP7 EU project United4Health (United4Health,

2015), currently being successfully used in 3 inter-

municipal telemedicine centres in Norway.

Future research would include a full

implementation of the system, with its

corresponding evaluation in the field from a

usability and operational perspective. In addition, a

comparison of the new and the already existing

system would provide useful results.

ACKNOWLEDGEMENTS

The authors thank all participants for their

contribution in the research study and Devoteam AS

for the development and implementation of the

system. Financial support was provided by the

Regional Research Fund, Agder, Norway (Grant

number 204119-2011).

REFERENCES

Bangor, A., Kortum, P., Miller, J., 2009. Determining

What Individual SUS Scores Mean: Adding an

Adjective Rating Scale. Journal of Usability Studies,

Vol. 4, Issue 3, 114-123.

Brooke, J., 1996. SUS: a quick and dirty usability scale.

Usability Evaluation in Industry. London: Taylor &

Francis, p.189-194.

HEALTHINF 2016 - 9th International Conference on Health Informatics

312

Chan, J., Shojania, K. G., Easty, A. C., Etchells, E. E.,

2011. Does user-centred design affect the efficiency,

usability and safety of CPOE order sets? Journal of

the American Medical Informatics Association, 18,

276e281. doi:10.1136/amiajnl-2010-000026.

Directorate of Health and National Expertise Service for

Ageing and Health, 2011. Etablering og drift av.

Demensteam, Demensutredning i primærhelsetjenesten.

[cited 2015 Dec 10]. Available from: http://

www.aldringoghelse.no/ViewFile.aspx?ItemID=2824.

Ericsson, K. A., Simon, H. A., 1980. Verbal reports as

data. Psychological Review, 87(3), 215-251.

Folstein, M. F., Folstein, S. E., McHugh, P. R., 1975.

“Mini-mental state”: A practical method for grading

the cognitive state of patients for the clinician. Journal

of Psychiatric Research, Vol 12, Issue 3, 189-198.

doi:10.1016/0022-3956(75)90026-6.

Gerdes, M., Smaradottir, B., Fensli, R., 2014. End-to-end

infrastructure for usability evaluation of ehealth

applications and services. Proceedings of

Scandinavian Conference on Health Informatics,

Grimstad, Norway, August 21-22, 53-59. ISSN(print):

1650-3686, ISSN(online): 1650-3740, ISBN: 978-91-

7519-241-3.

Goldberg, L., Lide, B., Lowry, S., Massett, H. A.,

O'Connell, T., Preece, J., Quesenbery, W.,

Shneiderman, B., 2011. Usability and accessibility in

consumer health informatics current trends and future

challenges. American Journal of Preventive Medicine

Vol 40, Issue 5, Supplement 2, S187–S197.

doi:10.1016/j.amepre.2011.01.009.

Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S.,

Persson J., Cajander, Å., 2003. Key principles for

user-centred systems design. Behaviour & Information

Technology, 22:6, 397-409.

doi:10.1080/01449290310001624329.

Gustafson, L., 1996. What is dementia? Acta Neurologica

Scandinavica, Volume 94, Issue Supplement s168,

22–24. doi: 10.1111/j.1600-0404.1996.tb00367.x.

Holen-Rabbersvik, E., Eikebrokk, T. R., Fensli, R. W.,

Thygesen, E., Slettebø, Å., 2013. Important challenges

for coordination and inter-municipal cooperation in

health care services: a Delphi study. BMC health

services research, 13(1), 451. doi: 10.1186/1472-

6963-13-451.

Jaspers, M. W., 2009. A comparison of usability methods

for testing interactive health technologies:

methodological aspects and empirical evidence.

International Journal of Medical Informatics, 78(5),

340-353.

Kushniruk, A. W., Patel, V. L., 2004. Cognitive and

usability engineering methods for the evaluation of

clinical information systems. Journal of Biomedical

Informatics, 37(1), 56-76.

Lazar, J. 2006. Web Usability- a user-centered design

approach. Pearson Education. ISBN-10: 0321321359

ISBN-13: 9780321321350.

Lazar, J., Feng, J. H., Hochheiser, H., 2010. Research

methods in human-computer interaction, Chichester,

United Kingdom, John Wiley & Sons, ISBN 978-0-

470-72337-1.

MacLellan, S., Muddimer, A., Peres S., C. 2012. The

Effect of Experience on System Usability Scale

Ratings. Journal of Usability Studies, Vol. 7, Issue 2,

56-67.

NHN, 2015: Norwegian Health Network. [cited 2015 Dec

10]. Available from: https://www.nhn.no/english/

Pages/default.aspx.

Nielsen, J., 1994. Estimating the number of subjects

needed for a thinking aloud test. International Journal

of Human-Computer Studies, 41(3), 385-397.

doi:10.1006/ijhc.1994.1065.

Nielsen, J. 1993. Usability engineering, Elsevier. ISBN-

13: 978-0125184069, ISBN-10: 0125184069.

Nielsen, J., and Landauer, T. K., 1993. A mathematical

model of the finding of usability problems.

Proceedings of ACM INTERCHI'93 Conference,

Amsterdam, The Netherlands, April 24-29, pp. 206-

213.

Norwegian Ministry of Health and Care Services, 2007.

Dementiaplan 2015 (Demensplan 2015 “Den gode

dagen” Delplan til omsorgsplan 2015). [cited 2015

Dec 10]. Available from: http://www.regjeringen.no/

upload/HOD/Vedlegg/Omsorgsplan_2015/Demenspla

n2015.pdf.

Norwegian Ministry of Health and Care Services, 2008-

2009. Report No. 47. The Coordination Reform,

Proper treatment – at the right place and right time.

[cited 2015 Dec 10]. Available from:

https://www.regjeringen.no/contentassets/d4f0e16ad3

2e4bbd8d8ab5c21445a5dc/no/pdfs/stm200820090047

000dddpdfs.pdf.

NSD, 2015: Norwegian Social Science Data Services.

[cited 2015 Dec 10]. Available from:

http://www.nsd.uib.no/personvern/en/index.html.

Smaradottir, B., Gerdes, M., Martinez, S., Fensli, R.,

2015a. The EU-project United4Health: user-centred

design of an information system for a Norwegian

telemedicine service. Journal of Telemedicine and

Telecare Nov 2015. doi:10.1177/1357633X15615048.

Smaradottir B., Holen-Rabbersvik, E., Thygesen, E.,

Fensli, R., Martinez, S., 2015b. User-centred design

and usability evaluation of the user interface of a

collaborative information system for inter-municipal

dementia team. Proceedings of International

Conference on Health Informatics, Lisbon, Portugal

January 12-15, 446-453. doi:

10.5220/0005222704460453.

Smaradottir B., Holen-Rabbersvik, E., Thygesen, E.,

Fensli, R., Martinez, S., 2014. Usability evaluation of

electronic forms and collaborative assessment report in

an inter-municipality health care team for dementia

diagnose. Proceedings of Scandinavian Conference on

Health Informatics, Grimstad, Norway, August 21-22,

45-54. ISSN(print): 1650-3686, ISSN(online): 1650-

3740, ISBN: 978-91-7519-241-3.

United4Health 2015, European Commission

Competitiveness Innovation Programme, ICT

Programme to Support Policy. [cited 2015 Dec 10].

Available from: http://www.united4health.eu.

Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process

313