Toward a Compare and Contrast Framework for COVID-19 Contact

Tracing Mobile Applications: A Look at Usability

Cristiano Storni

, Damyanka Tsvyatkova

, Ita Richardson

1,2 c

, Jim Buckley

1,2 d

Manzar Abbas

, Sarah Beecham

, Muslim Chochlov

1,2

, Brian Fitzgerald

, Liam Glynn

Kevin Johnson

, John Laffey

, Bairbre McNicholas

, Bashar Nuseibeh

, James O’Connell

Derek O’Keeffe

, Ian R O'Keeffe

, Mike O’Callaghan

, Abdul Razzaq

, Kaavya Rekanar

Andrew Simpkin

, Jane Walsh

and Thomas Welsh

University of Limerick, Republic of Ireland

The Science Foundation Ireland Research Centre for Software, Lero, University of Limerick, Republic of Ireland

National University of Ireland Galway, Republic of Ireland

Keywords: COVID-19, mHealth, Evaluation Taxonomy, Usability.

Abstract: This paper reports on the progress in the project COVIGILANT, which is aimed at developing an evaluation

taxonomy for Contact Tracing Applications (CTAs) for COVID-19. Specifically, this article describes the

development of Usability, one pillar of the COVIGILANT taxonomy, discussing the classification and

decision-making processes, and the initial model validation. The validation process was undertaken in two

stages. First, we validated how the Usability pillar could be used to evaluate the Irish Health Services

Executive (HSE) COVID-19 CTA. While this supported many of the attributes that we had within the

Usability pillar, it also identified issues. We made amendments based on these, and undertook a second study,

this time evaluating 4 CTAs used in other countries. This has led to the completion of the Usability pillar,

which can now be used to evaluate global CTAs.

1 INTRODUCTION

As the novel coronavirus (Covid-19) spreads across

the globe with limited treatments and no vaccine as

yet discovered, governments and public health

institutions look at a series of non-pharmaceutical

intervention strategies to limit the spread of the virus.

In this context, contact tracing (CT) is seen as a

promising strategy to identify, isolate and contain

outbreaks. Although traditionally a manual process,

digital contract tracing applications (CTA) have been

proposed to leverage the pervasive use of smartphone

devices and increase the accuracy of the CT process.

This paper reports on progress in the

‘COVIGILANT’ project, a project aimed at

developing a compare and contrast evaluation

framework for CTAs which would help to improve

https://orcid.org/0000-0003-1601-0900

https://orcid.org/0000-0002-1686-7413

https://orcid.org/0000-0002-5493-2837

https://orcid.org/0000-0001-6928-6746

existing solutions (Buckley et al., 2020). For this

purpose, the COVIGILANT project aims at better

understanding the population’s perception of Contact

Tracing Apps (O’Callaghan et al., 2020) in the Irish

context and to develop a taxonomy to compare and

contrast the different aspects of contact tracing apps

developed globally. The COVIGILANT taxonomy

currently includes 7 pillars which were developed by

the project team as aspects to be examined when

evaluating CTAs using compare and contrast:

Characteristics, Effectiveness, Performance, User

Autonomy/Self Determination, Data Protection,

Transparency and Usability. Specifically, within each

pillar, there are attributes, sub-attributes and a

corresponding set of questions which are posed

during CTA evaluation. In this paper, we focus on the

development of the Usability pillar of the

COVIGILANT taxonomy. We use literature to

Storni, C., Tsvyatkova, D., Richardson, I., Buckley, J., Abbas, M., Beecham, S., Chochlov, M., Fitzgerald, B., Glynn, L., Johnson, K., Laffey, J., McNicholas, B., Nuseibeh, B., O’Connell, J.,

O’Keeffe, D., O’Keeffe, I., O’Callaghan, M., Razzaq, A., Rekanar, K., Simpkin, A., Walsh, J. and Welsh, T.

Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability.

DOI: 10.5220/0010307005570565

In Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2021) - Volume 5: HEALTHINF, pages 557-565

ISBN: 978-989-758-490-9

557

develop the pillar, and illustrate its effectiveness by

using it to evaluate 5 existing CTAs which are being

used in different countries.

The first section provides a definition of COVID-

19 CTAs, their types, the international regulations for

their development and design, and limitations found

in the existing frameworks. The second section

reports on the development of the COVIGILANT

taxonomy, specifically its Usability pillar and the

structure of the attributes within. This section also

explains how we tested, reviewed and refined our first

iteration of the Usability pillar. The paper concludes

with a summary of the findings and outlines plans for

future work.

2 CTA FOR COVID-19:

DEFINITION, TYPES, EU

REGULATIONS AND

AVAILABLE FRAMEWORKS

Paper-based CT strategies have existed through

previous pandemics (Swanson et al., 2018). However,

these are intense and require tedious manual entry

which is prone to human error. The use of technology

for tracing activity has shown promising results in the

monitoring process of Ebola (Danquah et al., 2019)

and has generated understandable enthusiasm as a

means to prevent the further spread of COVID-19

(Goggin, 2020). In this section, we first provide a

definition of CTAs and their various types as they are

discussed in the growing academic literature on this

subject. We outline some of the regulations and

principles suggested for their design, discussing

existing frameworks offered for evaluating CTAs.

2.1 Definitions and Different Types of

CTAs

CTAs are mobile applications that are developed for

and used in smartphone technology. According to the

World Health Organisation (WHO) (2020), these

applications are “also known as proximity tracking

tools”, that “use location-based (GPS) or Bluetooth

technology to find and trace the movements of

individuals to identify people who may have been

exposed to an infected person”. Different types of

CTA are being developed. These use different

Transparency about the purpose, transparency about the

design choices, transparency about the benefits of using the

application, minimum amount of personal data collection,

protect users by using pseudonymous identifiers, ensure

approaches to contact tracing and different

approaches to data storage, management and usage.

To help classify these approaches, the WHO (2020)

considered two main forms of CTAs: a) centralised

form where the processing of the contact history data

is central, usually undertaken by health authorities

and b) decentralised where the data is kept in the

users’ devices. In both cases, GPS or Bluetooth

technologies are used to collect data and include alert

notification functions to warn users of a potentially

infectious exposure.

2.2 EU Regulations for the

Development and Design of CTAs

To develop effective app solutions, a series of design

requirements were proposed by the European

Commission to ensure that the use of the digital tool:

a) will address accessibility and inclusiveness and be

on a voluntary basis,

b) be approved by the national health authority to be

monitored, and

c) will include forms of encryption for preserving

personal data.

Additionally, once the pandemic is over, the app

can be deactivated and disassembled (eHealth

Network, 2020).

Similarly, the Information Commissioner's Office

(ICO) (2020) published ten core principles

to follow

when designing CTAs. These have been grouped to

cover the application development life cycle. Some of

these principles discuss the importance of building

“core requirements” which include “user experience

(UX) design and other appropriate in-app

transparency mechanisms”, and that they “should

adopt a user centric design approach” (ICO, 2020).

However, despite all these regulations, requirements

and principles, the CTAs will be effective only if they

are integrated into the “existing public health system”

and adopted by the majority of the population (WHO,

2020).

2.3 Existing Frameworks for CTAs and

Their Limitations

Our first steps were to look at the available academic

and grey publications discussing evaluation

frameworks for CTAs. A Cho et al. (2020) and De

Carli et al. (2020) offer two frameworks which

users have control over the app, store collected data for a

minimum amount of time as users have control over it,

encryption of data, voluntary participation and ensure

privacy and security.

HEALTHINF 2021 - 14th International Conference on Health Informatics

558

primarily focus on privacy issues. Gasser et al. (2020)

propose a framework whose scope is the ethical and

legal challenges of CTAs, while Dar et al. (2020) look

mainly at feasibility and effectiveness aspects.

Despite certain differences, these frameworks

commonly highlight the specific importance of

privacy and data protection in CTAs, and this led us

to acknowledge that we need a specific pillar about

this aspect in our taxonomy. Only the assessment

framework for mobile and web-based applications

developed for COVID-19 suggested by Vokinger et

al. (2020) includes usability aspects. Vokinger et al.’s

framework is based on “an existing trustworthiness

checklist for digital health applications” and

comprises eight domains (i.e., Purpose, Usability,

Information Accuracy, Transparency, Organisational

Attributes/Reputation, Privacy and User

Control/Self-Determination) (2020). The authors,

however, also highlighted the following limitations:

1) The framework was specifically designed to the

Swiss privacy law aspects which stress

transparency and user privacy but limits it to the

Swiss context, and

2) Usability criteria and technical characteristics of

CTAs are included but not developed sufficiently.

We have identified this as a research gap, and our

research question for this paper is:

How can Usability attributes be integrated into a

taxonomy through which CTAs can be compared,

contrasted and possibly improved?

3 RESEARCH METHOD:

CREATING AND ASSESSING

USABILITY PILLAR FOR CTA

To address the limited scope of the CTA frameworks

we initially reviewed with respect to Usability, and to

further develop a Usability pillar, three researchers

assessed three further types of literature in detail.

First, we looked at the most recent guidelines for

mobile apps (Alturki & Gay, 2019; Goel et al., 2018;

Shitkova et al., 2015; Weichbroth, 2020), and

specifically for mobile health apps (e.g., mHealth)

(Kasali et al., 2019; Kaur & Haghighi, 2016; Xcertia,

2019). This literature is important because it

overviews a number of Usability attributes and

guidelines. For instance, 75 attributes for the

Usability of mobile applications were distinguished

and analysed through a systematic literature review

by using only the Scopus database (Weichbroth,

2020). The article discussed only the attributes that

appeared more than once (N=24). Xcertia (2019)

provides guidelines on five key areas of Privacy,

Security, Usability, Operability and Content to ensure

that mobile health apps cover the needs of clinicians

and end-users. We also searched more traditional

literature on Usability including: the Usability

standards ISO 9241-11, Nielsen Usability framework

(Nielsen, 1993), Preece et al. (2015) Usability

taxonomy (1998/2019) and notions from Norman’s

psychology (Norman, 2013) of everyday objects

(1998/2012, e.g., Affordances, Constraints and

Mapping). This literature gave us a rich range of

notions to focus on for the development of a Usability

pillar (e.g., User Satisfaction is found in all mentioned

frameworks, Efficiency and Learnability are

mentioned on most of them). Eventually, we also

acknowledged the key importance of Accessibility

which is often included as an aspect of Usability and

which led us to further review the literature and

standards on Accessibility (Ballantyne et al., 2018;

ETSI, 2018; EU, 2016) and Universal Design (UD)

of mHealth apps (Harrington et al., 2017; Kascak et

al., 2014). That literature includes design

requirements for users with various type of

impairments (e.g., visual, motor, cognitive), but also

‘special’ categories of users such as the elderly. This

initial review helped us to identify an initial list of 53

attributes relevant to CTA, and Table 1 shows all the

used sources (first column) and attributes (second

column).

The initial list of attributes was discussed by a

subset of the COVIGILANT team (8 researchers),

and against the 7 initial pillars for the COVIGILANT

taxonomy: Characteristics, Usability, Effectiveness,

Performance, User Autonomy/Self Determination,

Data Protection and Transparency. We acknowledge

that some issues (which in a different context could

be treated under Usability) deserved a more specific

focus in CTAs. Based on this consideration, aspects

such as Effectiveness, Performance, User Autonomy

and Transparency were developed as pillars on their

own, independent from the Usability group, by other

members of the team. In line with existing CTA

frameworks, Privacy and Data Protection were also

treated independently.

Seven of the discussed attributes were specific to

Usability and were used as starting points for the

development of the Usability pillar (version 1.0)

(Figure 1). These were:

1) Subjective Satisfaction: “refers to how pleasant

it is to use the system” (Nielsen, 2009);

2) Availability (New): inequalities in the access

based on internet connectivity and platform

dependency;

Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability

559

Table 1: An overview of the most relevant attributes identified in academic publications.

Frameworks Proposed attributes

Contact tracing apps for COVID-19

(Vokinger et al., 2020)

Purpose, Usability, Information Accuracy, Organisational Attributes

/Reputation, Transparency, Privacy, User Control/Self-Determination

mHealth

(Kasali et al., 2019; Kaur & Haghighi, 2016;

Xcertia, 2019)

Efficiency, Satisfaction, Effectiveness, Learnability, Memorability,

Cognitive Load, Simplicity, Universality, Aesthetics,

Security, Usefulness, Resources, Troubleshooting,

Ongoing App Evaluation, Name of the Application

UD for mHealth apps for older individuals

(Harrington et al., 2017; Kascak et al., 2014)

Navigation, Affordances, Interaction, Equitable Use, Flexibility,

Ease of Use, Errors, Low Physical Effort,

Size and Space for Approach and Use

Accessibility

(Ballantyne et al., 2018)

Perceivable, Operability, Understandability, Robustness,

Design, System, Content

Mobile applications

(Alturki & Gay, 2019; Goel et al., 2018;

Shitkova et al., 2015; Weichbroth, 2020)



Attractiveness, Comprehensibility, Accessibility, Consistency,

Training, Trust, Battery Consumption, Less Storage Consumption,

Adaptability, Performance, Layout, Platform Dependency,

Onboarding, Speed, ISO 9241-11 (Efficiency, Satisfaction and

Effectiveness)

Figure 1: Visualisation of the initial structure of the Usability pillar (version 1.0).

3) Accessibility: design guidelines for individuals

with cognitive, visual, hearing and dysmotility;

4) Flexibility: allow users to perform tasks in

various ways to accelerate their performance;

5) Effectiveness: the ability of a user to complete a

task to achieve their goals;

6) Interaction: design of the User Interface (UI) and

user interactions;

7) Ongoing App Evaluation: presence of user

research and iterative evaluations after the app

was released.

We included Accessibility under Usability.

Furthermore, we found that Platform Dependency

and Internet Connectivity are also elements of

Accessibility to be accounted for as CTAs might not

work in all smartphones and typically require an

active connection to be installed during setup. To

accommodate this, we included an ‘Availability’

category under Accessibility. Satisfaction was

changed to ‘Subjective Satisfaction’ adhering to

Nielsen's definition (2009).

HEALTHINF 2021 - 14th International Conference on Health Informatics

560

After identifying these seven high-level

dimensions, our job was to again review the literature,

so that we could populate these high-level categories

with attributes, sub-attributes and specific questions

to be asked when using our Usability pillar to assess

existing CTAs. For each attribute and sub-attribute, a

definition was given (based on that literature) and at

least one probing question (also based on literature

when this is available, or freely formulated by us

otherwise). We included a commentary on the

feasibility of such questions, detailing how one could

possibly answer them (e.g., the Devil’s Advocate

approach). This ensured that, not only had we

identified probing questions, but they could be

answered. This resulted in the top seven attributes

being expanded into specific sub-attributes with 77

related questions. A subset of our results is shown in

Table 1. To validate the Usability pillar, we tested

initial list of questions by applying them to the Irish

Health Services Executive (HSE) ‘COVID Tracker’

contact tracing app.

3.1 First Assessment of Usability Pillar:

HSE App Assessment

Our initial draft on the Usability pillar was tested with

the Irish HSE app, and a series of issues both in the

structure and in the questions were immediately

apparent:

1. The usability structure looked quite unbalanced

and disproportionate as:

a) two attributes had many sub-attributes (i.e.,

Accessibility and Interaction)

b) three attributes had no sub-attributes (i.e.,

Flexibility, Effectiveness and Subjective

Satisfaction)

c) some of the attributes (i.e., Accessibility and

Interaction) had sub-attributes with the same

labels (i.e., User Interaction and User Interface)

2. A few of the attributes were found too broad as:

a) Effectiveness is discussed in two places; as a

pillar of the COVIGILANT taxonomy, and as

one of the attributes in the Usability pillar

b) Accessibility included three sub-attributes (i.e.,

Functional Performance Statements, User

Interface and User Interaction)

c) Subjective Satisfaction was found to be too

subjective; it required better-formulated

questions

3. Many of the attributes and their subsequent

questions helped to identify more than one issue,

indicating:

https://covidtracker.gov.ie/

a) wrong positions of the attributes (i.e.,

Accessibility and Flexibility) and/or questions

(i.e., needed to relocate the question in the sub-

attribute Low Physical Effort)

b) not well-formulated questions (i.e., question

used for the sub-attribute User Interaction)

4. Most of the questions were developed for the

Interaction attribute. This provoked further

discussions on criteria for inclusion and exclusion

based on the assessment approach (e.g., heuristic

evaluation, should some attribute in Usability be

task-based rather than general about the whole

CTA?)

Table 2 presents a short summary of the issues

found with the appropriate solutions.

To achieve a more balanced structure on Usability

(version 2.0), we address the issues uncovered by

testing version 1.0 with the Irish HSE app (Table 2),

but we also refined the taxonomy based on the work

of Alonso-Ríos et al. (2010). We found this work to

be very helpful. Their work overviews and compares

existing Usability taxonomies towards an integrative

framework offering an exhaustive description and

definition of Usability attributes. It offered the

opportunity to clarify our terminology (e.g., the

difference between clarity and consistency) and to

expand and rearrange our categories to develop a

hierarchical structure of Usability, removing

redundancy and repetitions. We made several

changes in the Usability structure and questions:

1. ‘Effectiveness’ and ‘Interaction’ attributes were

re-labelled respectively into Design Effectiveness

and User Interaction (Tasks Specific). Each of

these attributes has four sub-attributes. For the

former, these attributes are covering the capacity

of the system, interface, and interaction design

(i.e., Completeness, Configurability, User

Interface and Helpfulness). For the latter, they are

looking at the interaction with the app UI in

performing a particular task (i.e., Efficiency,

Robustness, Clarity of Interaction with Elements

and Consistency of Interaction with Elements).

2. We created two new sub-attributes for Subjective

Satisfaction (i.e., Motivations for High Score and

Motivations for Low Score).

3. Universality is a new attribute; it refers to the

ability of the app to be used by different users

(different because of impairments or culture); it

has two sub-attributes Accessibility and Cultural

Universality.

4. Questions were further refined in a bottom-up

fashion. This approach helped to identify a limited

number of specific questions related to the

Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability

561

attributes which were more effective for doing a

CTA review.

We undertook a second validation phase, using the

Usability pillar again to evaluate 4 further CTAs more

specifically (e.g., individual questions picked up

unique CTA issues), and categories and definitions

were clearer.

3.2 Four CTAs Assessments: Results

and Amendments to the Usability

Pillar

We identified four CTAs

with different characteristics

and used the Usability version 2.0 against them. These

CTAs were developed for health authorities and users

in EU and non-EU countries. They were selected on

the basis that they had different characteristics, for

example, content, user interface design and

navigational elements. This variety allowed us to test

the Usability pillar structure and questions to detect

where improvements are possible or needed. We found

these activities very useful in two ways. First, it

particularly helped to add/improve some of the

questions we had under Accessibility (i.e., for the input

fields and activation of accessibility technologies in the

device settings) and it helped in the Universality

attribute for the Language subsection. Second, it

helped to identify new sub-attributes to be added to the

pillar such as Notifications, Content and Age/Parental

Control (Figure 2).

Table 2: Examples of issues identified in the usability structure and questions (version 1.0) along with the appropriate

solutions.

Attribute/

Sub-attributes/Questions

Refinement issues Solutions

Attribute:

Effectiveness

a) Use the same label with a pillar examining

the effectiveness of contact tracing function

b) Does not include sub-attributes

a) Re-label as a Design Effectiveness

b) Create four sub-attributes, i.e., Completeness,

Configurability, User Interface and Helpfulness.

Attribute:

Accessibility

a) Clashing with Universality

b) Includes many sub-attributes (i.e.,

Functional Performance Statements, User

Interface and User Interaction) overlapping

with more traditional Usability issues

a) Create a new attribute Universality with two sub-

attributes, i.e., Accessibility and Cultural Universality

b) Create three sub-attributes for Accessibility, i.e.,

Functional Performance, UI Elements, Accessible

Interactions

Attribute:

Interaction

a) Include most of the questions developed for

the Usability pillar

b) Refers to the actual interaction with an

interface in the execution of specific tasks

a) Re-label as User Interaction (Tasks Specific)

b) Create four sub-attributes, i.e., Efficiency,

Robustness, Clarity of Interaction with Elements and

Consistency of Interaction with Elements

Sub-attribute:

User Interaction

a) Appears in two attributes, i.e., Accessibility

and Interaction

b) Refers to the actual interaction with an

interface in the execution of specific tasks

a) Create only one main attribute, i.e., User

Interaction (Tasks Specific) that includes four sub-

attributes, i.e., Efficiency, Robustness, Clarity of

Interaction with Elements and Consistency of

Interaction with Elements

Sub-attribute:

User Interface

a) Appears in two attributes, i.e., Accessibility

and Interaction

b) Refers to the User Interface elements and

how accessible they are designed

a) Re-label as UI Elements

b) Relocate the sub-attribute in the attribute

Universality, sub-attribute Accessibility

Question:

Q1: Is multimodality

offered?

a) Question assesses more than one issue, i.e.,

Accessibility and User Interaction

b) Current position - in the attribute

Interaction, sub-attribute User Interaction

c) Refers to the multiple modes available for

interaction, i.e., Flexibility/Multimodality

a) Create a new sub-attribute with a label, i.e.,

Flexibility/Multimodality, attribute User Interaction

(Tasks Specific)

b) Relocate the question in the sub-attribute

Flexibility/Multimodality,

c) Formulate a better question, i.e.,

Q1: Can the same tasks be executed in different

ways?

Question:

Q1: Can a user complete

a task without scrolling?

a) Current position - in the sub-attribute

Navigation, attribute Accessibility

b) Refers to the reduction in physical effort,

i.e., sub-attribute Low Physical Effort

a) Relocate the question in the sub-attribute Low

Physical Effort, sub-attributes Accessibility and

Accessible Interactions

AMAN https://amanapp.jo/en

Corona-Warn-App https://www.coronawarn.app/en/

NOVID https://www.novid.org/

PathCheck SafePlaces https://pathcheck.org/

HEALTHINF 2021 - 14th International Conference on Health Informatics

562

Figure 2: Visualisation of the final Usability pillar (version 2.0) after application to the four CTAs.

Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability

563

For example, to support individuals who have

various limitations and disabilities, some of the CTAs

use various options for the input fields used for self-

reporting of a positive COVID-19 diagnosis, e.g.,

manually entering the data, voice control and QR

code. Likewise, promoting accessibility, it was found

that all apps support more than one spoken language

where selection was either in the app during the

activation process, in the app settings or in the device

system when picking up the preferred language.

Questions were improved to cover and assess the

availability of such options in the apps. In addition, a

few questions were added to assess whether the app

design accommodates the OS-level changes from the

device settings (e.g., font, colour and contrast) and the

accessibility features after they are activated from the

device settings (e.g., screen readers). We also

acknowledge age restriction as an important aspect of

CTAs. Only the Irish HSE app offers age restrictions

during the activation process, the other CTAs we

analysed describe these restrictions in the section

'Terms of Use'. To frame this aspect, we followed the

guidelines for inclusiveness (eHealth Network,

2020), and we added a new subgroup 'Age/Parental

control'.

The implementation of these amendments

brought us to finalise version 2.0 of our Usability

pillar, which we will continue to assess both

internally, by addressing redundancies and relocation

of attributes within the 7 pillars, and externally, by

continuing to test the Usability structure and

questions with real CTAs to stress its scope and

limitation.

4 USABILITY PILLAR: FUTURE

WORK

We presented the research question: How can

Usability attributes be integrated into a taxonomy

through which CTAs can be compared, contrasted

and possibly improved? Our approach was to draw on

different sources for design requirements (health

apps, mobile apps, accessibility guidelines (noting

particularly the elderly who are an at-risk category for

COVID-19)). This top-down approach, drawing on

existing literature and helping us to identify 53 initial

attributes, was complemented with bottom-up,

iterative-refinement of Usability version 1.0.

Through an initial test using the Irish HSE COVID-

19 CTA, we identified both the questions that worked

and the limitations in the initial structure and

questions. We undertook further exploration of

available literature, expanding beyond healthcare,

and proposed a more balanced structure by adding

new attributes and formulating task-specific

questions up to a refined version 2.0 of our Usability

pillar. The same strategy was applied to the other 6

pillars (see Welsh et al., 2020, for work on Data

Protection pillar) included in the COVIGILANT

taxonomy. The next planned activities are focused on

an amalgamation of all pillars, the assessment of the

full set of questions again in order to propose a refined

evaluation framework that can effectively evaluate

various types of CTAs. We are considering how

different questions might concern and be relevant for

different stakeholders in the CTAs ecology, for

example, developers, Health Departments, data

protection authorities. We are exploring how different

‘vistas’ could be created to align with the different

concerns of different stakeholders, thus offering a

tailored subset of pertinent questions to understand

where CTAs could be improved.

ACKNOWLEDGEMENTS

This work was supported, in part, by Science

Foundation Ireland grant 13/RC/2094 and the

COVIGILANT Science Foundation Ireland grant

20/COV/0133.

REFERENCES

Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E.,

& Moret-Bonillo, V. (2010). Usability: A critical

analysis and a taxonomy. International Journal of

Human-Computer Interaction, 26(1), 53–74.

Alturki, R., & Gay, V. (2019). Usability Attributes for

Mobile Applications: A Systematic Review.

Ballantyne, M., Jha, A., Jacobsen, A., Scott Hawker, J., &

El-Glaly, Y. N. (2018). Study of accessibility

guidelines of mobile applications. ACM International

Conference Proceeding Series, 305–315.

https://doi.org/10.1145/3282894.3282921

Buckley, J., Abbas, M. M., Chochlov, M., Fitzgerald, B.,

Glynn, L., Johnson, K., Laffey, J., McNicholas, B.,

Nuseibeh, B., Connell, J., O’Keeffe, D., O’Keeffe, I. R.,

O’Callaghan, M., Razzaq, A., Rekanar, K., Richardson,

I., Simpkin, A., Storni, C., Tsvyatkova, D., … Welsh,

T. (2020). A Proposal for the SFI Covid-19 Rapid

Response Funding Call. https://tinyurl.com/y4qbvk8c

Cho, H., Ippolito, D., & Yu, Y. W. (2020). Contact Tracing

Mobile Apps for COVID-19: Privacy Considerations

and Related Trade-offs. ArXiv Preprint

Danquah, L. O., Hasham, N., MacFarlane, M., Conteh, F.

E., Momoh, F., Tedesco, A. A., Jambai, A., Ross, D. A.,

& Weiss, H. A. (2019). Use of a mobile application for

HEALTHINF 2021 - 14th International Conference on Health Informatics

564

Ebola contact tracing and monitoring in northern Sierra

Leone: A proof-of-concept study. BMC Infectious

Diseases, 19(1).

Dar, A. B., Lone, H., Zahoor, S., Amin Khan, A., & Naaz,

R. (2020). Applicability of Mobile Contact Tracing in

Fighting Pandemic (COVID-19): Issues, Challenges

and Solutions.

De Carli, A., Franco, M., Gassmann, A., Killer, C.,

Rodrigues, B., Scheid, E., Schoenbaechler, D., &

Stiller, B. (2020). WeTrace -- A Privacy-preserving

Mobile COVID-19 Tracing Approach and Application.

1–15. http://arxiv.org/abs/2004.08812

eHealth Network. (2020). Mobile applications to support

contact tracing in the EU’s fight against COVID-19:

Common EU Toolbox for Member States.

ETSI. (2018). ETSI EN 301 549 - V2.1.2 - Accessibility

requirements for ICT products and services. 2, 1–

152.https://www.etsi.org/deliver/etsi_en/301500_3015

99/301549/02.01.02_60/en_301549v020102p.pdf

EU. (2016). DIRECTIVE (EU) 2016/2102 OF THE

EUROPEAN PARLIAMENT AND OF THE

COUNCIL of 26 October 2016 on the accessibility of

the websites and mobile applications of public sector

bodies. Official Journal of the European Union, 327, 1–

15. https://eur-lex.europa.eu/eli/dir/2016/2102/oj

Gasser, U., Ienca, M., Scheibner, J., Sleigh, J., & Vayena,

E. (2020). Digital tools against COVID-19: Framing

the ethical challenges and how to address them.

Goel, S., Nagpal, R., & Mehrotra, D. (2018). Mobile

Applications Usability Parameters: Taking an Insight

View. In Mishra D., Nayak M., & Joshi A (Eds.),

Information and Communication Technology for

Sustainable Development (Vol. 9, pp. 35–43). Springer,

Singapore.

Goggin, G. (2020). COVID-19 apps in Singapore and

Australia: reimagining healthy nations with digital

technology. Media International Australia,

1329878X2094977.

Harrington, C. N., Ruzic, L., & Sanford, J. A. (2017).

Universally Accessible mHealth Apps for Older

Adults: Towards Increasing Adoption and Sustained

Engagement. International Conference on Universal

Access in Human-Computer Interaction, 3–12.

ICO. (2020). COVID-19 Contact tracing: data protection

expectations on app development.

https://ico.org.uk/media/for-organisations/documents/

2617676/ico-contact-tracing-recommendations.pdf

Kasali, F. A., Taiwo, O. O., Akinyemi, I. O., Alaba, O. B.,

Awodele, O., & Kuyoro, S. O. (2019). An Enhanced

Usability Model for Mobile Health Application. 17(2),

20–29. https://sites.google.com/site/ijcsis/

Kascak, L. R., Rebola, C. B., & Sanford, J. A. (2014).

Integrating Universal Design (UD) principles and

mobile design guidelines to improve design of mobile

health applications for older adults. Proceedings - 2014

IEEE International Conference on Healthcare

Informatics, ICHI 2014, 343–348.

Kaur, E., & Haghighi, P. D. (2016). A context-aware

usability model for mobile health applications. ACM

International Conference Proceeding Series, 181–189.

https://doi.org/10.1145/3007120.3007135

Nielsen, J. (1993). Usability engineering. Morgan

Kaufmann. https://doi.org/10.1201/b16768

Nielsen, J. (2009). What Is Usability? In C. Wilson (Ed.),

User Experience Re-Mastered: Your Guide to Getting

the Right Design. Morgan Kaufmann.

Norman, D. (2013). The design of everyday things: Revised

and expanded edition (2nd ed.). Basic Books.

O’Callaghan, M. E., Buckley, J., Fitzgerald, B., Johnson,

K., Laffey, J., McNicholas, B., Nuseibeh, B., O’Keeffe,

D., O’Keeffe, I., Razzaq, A., Rekaner, K., Richardson,

I., Simpkin, A., Storni, C., Tsvyatkova, D., Walsh, J.,

Welsh, T., & Glynn, L. (2020). A National Survey of

attitudes to COVID-19 Digital Contact Tracing in the

Republic of Ireland. https://doi.org/10.21203/rs.3.rs-

40778/v1

Preece, J., Sharp, H., & Rogers, Y. (2015). Interaction

design: beyond human-computer interaction. John

Wiley & Sons. Chicago.

Shitkova, M., Holler, J., Heide, T., Clever, N., & Becker, J.

(2015). Towards Usability Guidelines for Mobile

Websites and Applications. Wirtschaftsinformatik,

1603–1617.

Swanson, K. C., Altare, C., Wesseh, C. S., Nyenswah, T.,

Ahmed, T., Eyal, N., Hamblion, E. L., Lessler, J.,

Peters, D. H., & Altmann, M. (2018). Contact tracing

performance during the Ebola epidemic in Liberia,

2014-2015. PLoS Neglected Tropical Diseases, 12(9).

https://doi.org/10.1371/journal.pntd.0006762

Vokinger, K. N., Nittas, V., Witt, C. M., Fabrikant, S. I., &

von Wyl, V. (2020). Digital health and the COVID-19

epidemic: an assessment framework for apps from an

epidemiological and legal perspective. Swiss Medical

Weekly, 150(1920), w20282.

Weichbroth, P. (2020). Usability of mobile applications: A

systematic literature study. IEEE Access, 8, 55563–

55577.

Welsh, T., Rekanar, K., Abbas, M., Chochlov, M.,

Fitzgerald, B., Glynn, L., Johnson, K., Laffey, J.,

McNicholas, B., Nuseibeh, B., O’Connell, J., O’Keeffe,

D., O’Keeffe, I. R., O’Callaghan, M., Razzaq, A.,

Richardson, I., Simpkin, A., Storni, C., Tsvyatkova, D.,

… Buckley, J. (2020). Towards a Taxonomy for

Evaluating Societal Concerns of Contact Tracing Apps.

To Appear In 2020 7th International Conference on

Behavioral, Economic, and Socio-Cultural Computing

(BESC).

WHO. (2020). Digital tools for COVID-19 contact tracing:

annex: contact tracing in the context of COVID-19, 2

June 2020. June. https://www.who.int/publicati

ons/i/item/WHO-2019-nCoV-Contact_Tracing-Tools_

Annex-2020.1

Xcertia. (2019). Xcertia mHealth App Guidelines. Xcertia.

https://www.himss.org/sites/hde/files/media/file/2020/

04/17/xcertia-guidelines-2019-final.pdf

Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability

565