Validation and Reﬁnement of Usability Heuristics for Interactive Web

Maps

Juliana Orro Marquez

1 a

, Paulo Meirelles

2 b

and Tiago Silva da Silva

1 c

Institute of Technology, Federal University of S

ao Paulo, S

ao Jos

e dos Campos, Brazil

Institute of Mathematics and Statistics of the University of S

ao Paulo, S

ao Paulo, Brazil

Keywords:

Usability Heuristic, Interactive Map, Information Visualization.

Abstract:

The usability of interactive web mapping systems is crucial for a wide range of applications, extending from

urban planning to personal navigation. This study, grounded in the context of Human-Computer Interaction

(HCI), speciﬁcally aims to enhance the usability of interactive web maps. The goal is to provide designers and

developers with improved guidelines that not only elevate the user experience but also effectively address the

unique challenges of these interfaces, promoting more efﬁcient navigation. Our methodology is distinguished

by the development of new usability heuristics, derived from a detailed analysis of the speciﬁcities of interac-

tive web mapping systems. The study proposes the introduction of a set of 12 usability heuristics, carefully

adapted for these systems. The preliminary results are promising, outlining a set of heuristics that have the

potential to be signiﬁcant in the design and implementation of interactive web maps. The contribution of this

study is substantial, offering new perspectives for the continuous improvement of usability heuristics and em-

phasizing the need for speciﬁc approaches for different digital interaction contexts. Thus, this work not only

advances the theoretical ﬁeld of HCI but also provides crucial practical guidelines for the future development

of interactive web mapping systems, meeting the current demands and expectations of users.

1 INTRODUCTION

The User Interface (UI) serves as the gateway through

which the user interacts with the system. This inter-

action underpins the overall user experience, which

may determine the effectiveness, efﬁciency, and sat-

isfaction during product use. A well-designed user

interface allows users to navigate and operate the sys-

tem in a planned manner, signiﬁcantly contributing to

a positive experience. Therefore, UI design is not just

about aesthetics; it is an essential component that con-

nects the user to the system, facilitating a harmonious

and efﬁcient interaction.

To ensure the efﬁciency of an interface design in

the Human-Computer Interaction (HCI) ﬁeld, qual-

ity of use criteria such as usability, communicability,

user experience, and accessibility are essential (Bar-

bosa et al., 2021). Among these, usability emerges

as a fundamental criterion for evaluating the effec-

tiveness of this interaction. According to Nielsen, us-

https://orcid.org/0000-0003-1013-7994

https://orcid.org/0000-0002-8923-2814

https://orcid.org/0000-0001-8459-7833

ability is deﬁned by the ease with which the interface

can be learned and used, as well as the satisfaction

provided by the system’s efﬁcient operation (Nielsen,

1994). To achieve this objective, principles known

as usability heuristics are employed, which guide de-

signers and developers in creating and evaluating in-

terface designs.

Since their formulation in the 1990s, Nielsen’s

heuristics have been a milestone in the ﬁeld of us-

ability (Nielsen, 1994). Composed of 10 widely rec-

ognized principles, these heuristics are used to com-

prehensively identify usability problems in interfaces.

However, despite providing a solid foundation, these

heuristics do not fully cover the nuances and speciﬁc

requirements associated with new devices and emerg-

ing systems, such as web-based interactive map vi-

sualization systems (Grifﬁn et al., 2017)(Roth et al.,

2017). This scenario highlights the need to adapt

or develop new heuristics that consider the unique

characteristics of these modern technologies (Jim

enez

et al., 2012).

The development of specialized usability heuris-

tics is vital for signiﬁcantly improving the user experi-

ence, particularly in areas where interactions are intri-

402

Marquez, J., Meirelles, P. and Silva da Silva, T.

Validation and Reﬁnement of Usability Heuristics for Interactive Web Maps.

DOI: 10.5220/0012691300003690

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 26th International Conference on Enterprise Information Systems (ICEIS 2024) - Volume 2, pages 402-409

ISBN: 978-989-758-692-7; ISSN: 2184-4992

cately complex, and demands are extremely speciﬁc.

In this context, this research proposes a new set of us-

ability heuristics meticulously adapted for interactive

maps on the web. Our goal is to provide designers and

developers with an enhanced set of guidelines that can

be applied to enrich the user experience in interact-

ing with these maps. This new list of heuristics aims

to address interactive maps’ peculiarities and unique

challenges, ensuring more efﬁcient and effective nav-

igation.

2 BACKGROUND

In the ﬁeld of HCI, usability heuristics play a crucial

role in the system evaluation process. They are used

as guidelines to identify potential usability problems.

Among the evaluation methods used in HCI, which

Barbosa et al. (2021) categorizes into three groups:

investigation, inspection, and observation, usability

heuristics are speciﬁcally employed during the in-

spection method, particularly in heuristic evaluation.

This approach involves a detailed analysis of inter-

faces to detect and resolve issues, thereby optimizing

the user experience. This technique involves a thor-

ough analysis of interfaces to identify and solve prob-

lems, signiﬁcantly contributing to the enhancement of

the user experience. Heuristic evaluation, a method-

ology introduced by Nielsen and Molich (1990), is

recognized as a highly effective inspection method,

characterized by its detailed analysis of products or

interfaces in search of usability issues. This approach

is valued for its agility, cost-effectiveness, and efﬁ-

ciency. During heuristic evaluation, specialized eval-

uators inspect a system’s interface, using a set of us-

ability heuristics as a reference for identifying prob-

lems, thereby ensuring a rigorous and comprehensive

assessment.

Nielsen’s heuristics, widely recognized in the ﬁeld

of usability, consist of ten fundamental principles,

each accompanied by a name and a detailed descrip-

tion (Nielsen, 1994):

Visibility of System Status (1). The design should

always keep users informed about what is happen-

ing within a reasonable timeframe.

Match Between the System and the Real World

(2). The design should speak the users’ language.

Use words, phrases, and concepts familiar to the

user, rather than internal jargon. Follow real-

world conventions, making information appear in

a natural and logical order.

User Control and Freedom. Users Often Perform

Actions by Mistake (3). They need a clearly

marked “emergency exit” to leave an unwanted

action without having to go through a lengthy pro-

cess.

Consistency and Standards (4). Users should not

have to wonder if words, situations, or actions

mean the same thing. Follow platform and indus-

try conventions.

Error Prevention (5). Eliminate operation errors

through conﬁrmation conditions and presentation

before proposing a conﬁrmation action.

Recognition Rather than Recall (6). Minimize the

user’s memory load by making elements, actions,

and options visible. The user should not have to

remember information from one part of the inter-

face to another. Information needed to use the de-

sign should be visible or easily retrievable when

needed.

Flexibility and Efﬁciency (7). Shortcuts, hidden

from novice users, can speed up interaction for

the experienced user, so that the design can cater

to both inexperienced and experienced users. Al-

low users to customize frequent actions.

Aesthetic and Minimalist Design (8). Interfaces

should not contain irrelevant or rarely needed in-

formation. Each extra unit of information in an

interface competes with relevant information units

and diminishes their relative visibility.

Help Users Recognize, Diagnose, and Recover

From Errors (9). Error messages should be ex-

pressed in plain language (no error codes), pre-

cisely indicate the problem, and constructively

suggest a solution.

Help and Documentation (10). It’s best if the

system does not need any additional explanation.

However, it may be necessary to provide docu-

mentation to help users understand how to com-

plete their tasks.

However, despite the proven effectiveness of these

general heuristics, they may not be fully suitable to

address the speciﬁc challenges found in new applica-

tions and devices. This gap has motivated the devel-

opment of new sets of heuristics tailored to speciﬁc

domains to meet the unique demands of these mod-

ern applications, as mobile maps (Kuparinen et al.,

2013), virtual worlds (Rusu et al., 2011), interactive

digital television (Solano et al., 2011), visualization

systems (Victorelli and Reis, 2020), and extended re-

ality applications (Vi et al., 2019),

With the aim of identifying new sets of usability

heuristics, Jimenez et al. (2016) conducted a litera-

ture review and identiﬁed nine articles proposing new

sets. However, it is important to note that despite the

Validation and Reﬁnement of Usability Heuristics for Interactive Web Maps

403

introduction of these new sets, they were not created

using a speciﬁc methodology designed for this pur-

pose. Instead, the majority adopted an approach that

combined Nielsen’s heuristics with domain-speciﬁc

characteristics, while others were developed based on

expert opinions. This varied approach to creating

heuristic sets highlights the lack of a standard in this

research area and the importance of adapting heuris-

tics to meet the speciﬁc needs of different application

domains.

To ﬁll these gaps, Qui

nones and Rusu (2019) pro-

posed a new methodology for the development of

domain-speciﬁc usability heuristics. This methodol-

ogy consists of eight stages, whose data inputs are:

• Exploration: collection of information about the

application, gathering usability attributes, and sur-

vey of a set of heuristics and/or other relevant el-

ements.

• Descriptive: Analyze data that are obtained in dif-

ferent experiments to collect additional informa-

tion that has not been identiﬁed in the previous

stage

• Description: classiﬁcation of selected information

and resources about the application, selection of

usability attributes, selection of a set of heuristics

and/or other relevant elements.

• Correlation: correlation of corresponding charac-

teristics, attributes, and existing heuristics, deﬁni-

tion of categories.

• Selection: process of adjusting and reﬁning infor-

mation for the creation of each heuristic.

• Speciﬁcation: description of the heuristics of the

new set.

• Validation: veriﬁcation of the performance of the

new set.

• Reﬁnement: descriptive document of the pro-

posed new set.

It provides a structured protocol to guide the pro-

cess of creating usability heuristics, ensuring their rel-

evance and effectiveness in speciﬁc domain contexts.

This approach has proven versatile in developing sets

of heuristics for various applications, such as Progres-

sive Web Applications (PWA) (Anuar and Othman,

2022) and Mobile Health Applications (UGmHA)

(Nasr et al., 2023).

In the context of interactive web maps, Marquez

et al. (2021b) identiﬁed the need to develop a speciﬁc

set of heuristics aimed at enhancing usability in this

area. To this end, they adopted the methodology pro-

posed by Qui

nones and Rusu (2019) to formulate a

new set of domain-speciﬁc usability heuristics. After

completing the stages of exploration (1), description

(3), correlation (4), selection (5), and speciﬁcation

(6), the authors introduced a set consisting of 10 spe-

ciﬁc heuristics for Interactive Web Mapping (IWM)

applications.

To validate this new set, Marquez et al. (2021a)

carried out the validation stage (7) of the methodol-

ogy by Qui

nones and Rusu (2019), which involves

three types of evaluation: expert analysis, heuristic

evaluation, and user testing. Initially, an expert analy-

sis was conducted, including a detailed evaluation of

each heuristic, focusing on clarity, ease of use, and the

need for additional checklists, to ensure that they ef-

fectively addressed the key usability aspects relevant

to IWM. The completeness of the heuristic set was

also assessed to ensure that it covered all critical ele-

ments affecting the user experience in interactive web

map applications. As a result of this process, Mar-

quez et al. (2021a) presented version 2 of the heuristic

set, incorporating reﬁnements based on expert feed-

back, aiming to further enhance its applicability and

effectiveness in the context of IWM. Subsequently,

a heuristic evaluation was conducted, the outcomes

of which led to the development of version 3 of the

heuristic set, detailed in this article.

3 RESEARCH METHOD

Continuing the quest for improvements, a new vali-

dation phase was conducted, heuristic evaluation, re-

sulting in the third version of the set of 12 usability

heuristics designed for IWM Applications. This itera-

tion represents an ongoing effort to reﬁne and expand

usability heuristics, ensuring that they more precisely

meet the demands of users in this speciﬁc context.

3.1 Data Collection

The heuristic evaluation was conducted with the pur-

pose of assessing the effectiveness of the proposed

heuristics in identifying usability issues. For this pur-

pose, both the speciﬁc set for IWM applications (Mar-

quez et al., 2021a) and Nielsen’s set (Nielsen, 1994)

were employed. The evaluation took place on the

CulturaEduca

platform, an innovative platform de-

veloped by Instituto Lidas in collaboration with the

Special Secretariat of Culture (SECULT/Brazil). This

platform provides a specialized tool for visualizing

geographic data and integrating essential information

in the ﬁelds of education and culture.

The evaluation involved the participation of eight

volunteers, comprising six males and two females.

Link: https://culturaeduca.cc/

ICEIS 2024 - 26th International Conference on Enterprise Information Systems

404

The volunteers included graduate students and pro-

fessionals in the ﬁeld of Computer Science, all of

whom had no prior experience using the platform.

This study received approval from the Ethics and Re-

search Committee of the Federal University of S

Paulo (CAAE: 52578921.4.0000.5505), and all par-

ticipants agreed to the Informed Consent Form (ICF)

presented at the beginning of the heuristic evaluation

process.

The evaluation procedure was divided into three

distinct stages:

• In the ﬁrst stage, we introduced the study and pre-

sented an explanatory video lasting twelve min-

utes detailing the theme and the nature of the eval-

uation.

• In the second stage, participants ﬁlled out a form

providing information about their previous experi-

ences in usability and geographic information vi-

sualization, characterizing the proﬁle of the vol-

unteers.

• Finally, in the third stage, participants con-

ducted the heuristic evaluation, identifying usabil-

ity problems on the platform during a 40-minute

period. We established this duration to balance

the depth and breadth of the evaluation with the

well-being and attention of the participants, based

on previous experience. In this period, 10 min-

utes are dedicated to familiarizing with the inter-

face, and another 10 minutes are allocated for spe-

ciﬁc tasks to obtain targeted insights on usability

aspects. The remaining 20 minutes are reserved

for free exploration, allowing for a more natural

and spontaneous interaction with the system. The

heuristic evaluation involved performing three ex-

ploration activities:

– Initially, participants conducted a search for

layers.

– Next, they performed a search for ﬁlters.

– Lastly, they had the remaining time to conduct

free exploration and collect usability problems.

The participants were randomly divided into two

groups: Group A was responsible for conducting the

evaluation using the proposed set of usability heuris-

tics for IWM (Marquez et al., 2021a), and Group B

used Nielsen’s usability heuristics (Nielsen, 1994).

3.2 Analysis

In total, 56 usability problems were identiﬁed, 32

identiﬁed by Group A and 24 by Group B. To assess

the performance of each set of heuristics, the ﬁve cri-

teria described in the methodology by Qui

nones and

Rusu (2019) were applied:

• Criteria 1 - Numbers of Correct and Incorrect As-

sociations of Usability Problems in Relation to

Heuristics:

The identiﬁed usability problems are analyzed

and classiﬁed as correct or incorrect in relation

to the heuristics they were associated with. The

effectiveness of Correct Associations (CA) and

Incorrect Associations (IA) was evaluated using

equations 1 and 2.

CA =

∑

n−1

CAHn

T P

× 100 (1)

IA =

∑

n−1

IAHn

T P

× 100 (2)

In this equation:

CA: Correct associations;

IA: Incorrect associations;

T: Total number of heuristics in the set;

CAHn: Number of correct associations of prob-

lems in ”n” heuristics;

IAHn: Number of incorrect associations of

problems in ”n” heuristics;

TP: Total usability problems identiﬁed.

To be considered as having good performance, the

experimental set should have CA(A) > CA(B) and

IA(A) < IA(B).

• Criteria 2 - Number of usability problems identi-

ﬁed:

Usability problems were divided into three

groups:

– P1 - Problems identiﬁed by both sets.

– P2 - Problems identiﬁed only by the experimen-

tal set (A).

– P3 - Problems identiﬁed only by the control

group (B).

The experimental set is considered to have good

performance if P1 and/or P2 are greater than P3.

• Criteria 3 - Number of usability problems identi-

ﬁed as speciﬁc:

To assess the Effectiveness of Usability problems

considered Speciﬁc (ESS), the selection criterion

adopted was problems directly related to the visu-

alization of geographic data. The efﬁcacy of the

number of speciﬁc problems was calculated using

the equation 3:

ESS =



NSP

T P



× 100 (3)

In this equation:

Validation and Reﬁnement of Usability Heuristics for Interactive Web Maps

405

ESS: Effectiveness;

NSP: Number of usability problems identiﬁed

as speciﬁc;

TP: Total usability problems identiﬁed.

To be considered as having good performance in

this criterion, it is necessary for ESS(A) to be

greater than ESS(B).

• Criteria 4 - Number of usability problems identi-

ﬁed that qualify as most severe:

Usability issues were assessed and categorized ac-

cording to their severity using Nielsen’s severity

scale (Nielsen, 1993) (Table 1). The Effectiveness

(ESV) of the set was determined by dividing the

number of usability problems with a severity rat-

ing greater than 2 by the total number of usability

problems (Equation 4).

ESV =



NPV

T P



× 100 (4)

In this equation:

ESV: Effectiveness;

NPV: Number of speciﬁc usability problems

identiﬁed that qualify as severity greater than

TP: Total usability problems identiﬁed.

Similarly to the previously mentioned criteria, the

performance of the experimental set (A) is consid-

ered good when ESV(A) is greater than ESV(B).

Table 1: Severity of usability problems.

Value Severity Description

0 No problem

Not considered a

usability problem.

Cosmetic

problem

No immediate need

for a solution, only if

there is extra time

available

Minor

problem

Fixing this problem

is desirable, but it

receives low priority

Major

problem

Fixing this problem

is desirable, but it

receives low priority

Catastrophic

problem

Important to ﬁx,

receives high priority

• Criteria 5 - Number of usability problems identi-

ﬁed that qualify as most critical:

Based on the number of usability problems ob-

tained in the severity analysis, we were able to

calculate the frequency (Equation 5) of the iden-

tiﬁed problems. Subsequently, we identiﬁed the

corresponding frequency value in Table 2 and cal-

culated the criticality by adding this value to the

severity of the problems:

Frequency =



gravity

T P



× 100 (5)

In this equation:

Frequency: Frequency of incidence;

Severity:Number of usability problems found

in the severity analysis;

TP: Identiﬁed usability problems.

Table 2: Value associated with the frequency of usability

problems for determining criticality.

Frequency Gravity

<1 0

1 - 10 1

11-50 2

51 - 90 3

>90 4

3.3 Results

The Table 3 presents the results of experimental group

A and control group B for each of the criteria men-

tioned earlier.

Table 3: Values achieved for the ﬁve performance criteria.

Criteria Group A Group B

Percentage of correct

associations (CA)

84,4 % 83,3%

Percentage of incorrect

associations (IA)

15,6% 16,7%

Number of usability

problems identiﬁed

P1 = 16

P2 = 22

P3 = 18

Efﬁcacy in terms of the

number of speciﬁc

usability problems

identiﬁed (ESS)

59,4% 62,5%

Efﬁcacy in terms of the

number of usability

problems that qualify as

more severe (ESV)

84,4% 54,2%

Efﬁcacy in terms of the

number of usability

problems that qualify as

most critical (ESC)

84,4% 54,2%

Based on the results, it becomes evident that the

differences in performance between Group A and

Group B raise interesting questions about the effec-

tiveness and speciﬁcity of usability heuristics in the

ICEIS 2024 - 26th International Conference on Enterprise Information Systems

406

context of interactive web maps. The ability of Group

A to identify a greater number of usability problems

suggests that the new heuristics created may have

value in addressing a wider range of issues. This

highlights the potential of these new heuristics to pos-

itively contribute to usability assessments in this do-

main.

On the other hand, Group B’s superior perfor-

mance in criterion 3, related to speciﬁc usability prob-

lems, indicates that there may be room for improve-

ments in the heuristics associated with this criterion.

This ﬁnding emphasizes the importance of reﬁning

and adapting heuristics to effectively address domain-

speciﬁc challenges.

To provide a qualitative overview, we delved into

the nature of the usability issues identiﬁed by each

group. For instance, the ﬁndings from Group A may

encompass a diverse range of issues spanning naviga-

tion, information layout, and user interaction, reﬂect-

ing the comprehensive applicability of the new heuris-

tics. In contrast, the problems identiﬁed by Group B,

are more specialized, indicating the need for further

reﬁnement of the heuristics that focus on speciﬁc us-

ability challenges.

While the differences are statistically detectable,

their magnitude is small and should be considered

in the appropriate context. Moreover, we acknowl-

edge that these subtle variations might not be robust

enough to directly inﬂuence design decisions or rec-

ommended practices without a deeper analysis of the

speciﬁc contextual nuances of use. This understand-

ing is crucial to ensure that our conclusions are both

accurate and relevant. Based on the ﬁndings, we were

able to make signiﬁcant adjustments to the proposed

heuristics to address the identiﬁed issues, reﬁning our

contribution to the usability ﬁeld.

4 USABILITY HEURISTICS FOR

INTERACTIVE WEB MAPS

After reﬁning the heuristics, we identiﬁed common-

alities and patterns that emerged in their application.

These insights have allowed for an organized group-

ing of the heuristics based on their shared attributes.

By categorizing them into cohesive groups, we en-

hance the clarity and ease of application, thereby in-

creasing their efﬁcacy in assessing the usability of in-

teractive web map systems. In this evolved frame-

work, the usability heuristics are strategically divided

into three targeted groups: map veriﬁcation heuristics,

information veriﬁcation heuristics, and system veriﬁ-

cation heuristics (Figure 1). Each group is meticu-

lously tailored to address the distinctive elements of

interactive web map systems. Every heuristic within

these groups is deﬁned with precision, assigned a

unique identiﬁer, and described with a lucid deﬁni-

tion, serving as an extensive toolkit for the evaluation

and improvement of interactive web map usability.

Figure 1: Representation of the three usability heuristics

groups for interactive web maps. Image created by the au-

thor.

4.1 Map Veriﬁcation Heuristics

In the Map Veriﬁcation Heuristics group, the heuris-

tics are speciﬁcally focused on the map area, meaning

they should be applied to assess the regions where one

or more maps are represented. The Figure 2 presents

examples of two usability problems identiﬁed by the

usability heuristics for interactive web maps.

Figure 2: Examples of usability problems identiﬁed by the

map veriﬁcation heuristic group. Image created by the au-

thor.

Representation of Geographic Data (M1). The in-

formation displayed on the map, such as symbols,

labels, or toponyms, should be visible regardless

of the scale adopted.

Map Manipulation Tools (M2). The map manipu-

lation tools should be functional, understandable,

and easily accessible.

Visualization of Geographic Information (M3).

The map layout should remain visible throughout

its use.

Scale of Representation (M4). The user should be

Validation and Reﬁnement of Usability Heuristics for Interactive Web Maps

407

informed about the scale at which the map is being

represented.

Cartographic Conventions (M5). The adopted

symbols should comply with cartographic stan-

dards and be presented in a visible and well-

deﬁned manner.

4.2 Information Veriﬁcation Heuristics

Concentrated on the presentation and organization of

information within the system, this set of heuristics

aims to ensure that data, both geographic and non-

geographic, is displayed in a logical, understandable,

and accessible manner. The Figure 4 presents an ex-

ample of a usability problem found with a heuristic

from the information veriﬁcation group.

Figure 3: Examples of usability problems identiﬁed by the

information veriﬁcation heuristic group. Image created by

the author.

Presentation of Geographic Information (M6).

The search results related to a position in geo-

graphical space should be organized and standard-

ized in their descriptions.

Presentation of Non-Geographic Information

(M7). The presentation of non-geographic infor-

mation should not overlap the entire map area.

Search Mechanisms (M8). The search mechanisms

should be prominently featured in the interface.

Export of Geographic Information (M9). Provide

a function that allows for the export of the map

and the researched geographic data.

4.3 System Veriﬁcation Heuristics

This focuses on the overall usability of the system and

the user’s interaction experience with the interface.

This group of heuristics addresses cross-device com-

patibility and the overall ease of use of the platform.

The Figure 4 presents an examples of usability ﬂaws

identiﬁed by the system veriﬁcation heuristic group.

Ease of Learning (M10). The interface should fa-

cilitate understanding of the control devices and

how to use them.

Figure 4: Examples of usability problems identiﬁed by the

system veriﬁcation heuristic group. Image created by the

author.

Interactivity (M11). The level of interactivity al-

lowed with the map should be clear to the user.

Compatibility with Different Devices and

Browsers (M12). The interface should adapt

to different screen sizes and offer the same

effectiveness regardless of the device or browser

used.

5 CONCLUSION

This study originated from the need to develop spe-

ciﬁc usability heuristics to meet the unique demands

of interactive web maps, recognizing the gap in gen-

eral usability heuristics. We focused our efforts on

creating a set of heuristics specially tailored for this

domain. The development of this set followed an

established protocol, based on a robust and proven

methodology. This systematic process not only en-

sured the comprehensiveness and relevance of the

heuristics but also ensured that they were meticu-

lously adapted to address the complexities and spe-

ciﬁc challenges found in interactive web map sys-

tems.

The development and reﬁnement of version 3 of

usability heuristics for interactive web maps represent

a signiﬁcant milestone in the reﬁnement process. The

ﬁrst version underwent a rigorous validation phase in-

volving experts in usability and cartography, which

was essential to ensure the accuracy and effectiveness

of the heuristics in the practical context of IWM sys-

tems. The adoption of an interdisciplinary approach

involving experts in both Computer Science and Car-

tography has proven to be extremely valuable, sig-

niﬁcantly expanding the scope and relevance of the

heuristics. This collaborative effort has the potential

to further advance the ﬁeld of usability evaluation in

the context of interactive web maps.

As presented in this article, the second version of

the heuristics was subsequently applied in the heuris-

tic evaluation of a geographic information visualiza-

ICEIS 2024 - 26th International Conference on Enterprise Information Systems

408

tion system, providing valuable insights for the de-

velopment of version 3. The results demonstrated su-

perior performance of the proposed set of heuristics

compared to the already established set. However,

the difference in the values found was not signiﬁcant,

highlighting the importance of heuristics adaptation.

This ongoing cycle of development, testing, and re-

ﬁnement underscores the evolving nature of heuris-

tics and their ability to adapt to changes and advance-

ments in the ﬁeld of interactive web maps.

This is an ongoing research, and our plans for fu-

ture work include several important steps. Firstly,

we plan to conduct a new expert analysis, followed

by another heuristic evaluation. Both analyses aim

to ensure the validity and comprehensiveness of the

developed usability heuristics. Additionally, we in-

tend to perform usability tests with real users, ensur-

ing that all tests receive equal attention and empha-

sis. To achieve this, we will form an interdisciplinary

team of participants, involving professionals from the

ﬁelds of Computer Science and Cartography, to en-

sure a comprehensive and representative evaluation.

These tests will allow for the practical validation of

the heuristics and the identiﬁcation of potential im-

provements. Based on the results of these tests, we

plan to iterate on the heuristics, reﬁning and adapt-

ing them as needed. Furthermore, we will continue

to collaborate with experts from different disciplines

to further enrich our approach and promote advance-

ments in usability evaluation in the context of interac-

tive web maps.

ACKNOWLEDGEMENTS

The authors would like to thank the participants for

their expertise and assistance throughout all aspects

of our study and the S

ao Paulo Research Foundation.

Grant 2021/06984-9, S

ao Paulo Research Foundation

(FAPESP).

REFERENCES

Anuar, N. N. and Othman, M. K. (2022). Development

and validation of progressive web application usabil-

ity heuristics (pwauh). Universal Access in the Infor-

mation Society, pages 1–29.

Barbosa, S. D. J., Silva, B. S. d., Silveira, M. S., Gas-

parini, I., Darin, T., and Barbosa, G. D. J. (2021).

Interac¸

ao Humano-Computador e Experi

encia do

Usu

ario. Autopublicac¸

ao.

Grifﬁn, A. L., Robinson, A. C., and Roth, R. E. (2017).

Envisioning the future of cartographic research.

Jimenez, C., Lozada, P., and Rosas, P. (2016). Usability

heuristics: A systematic review. In 2016 IEEE 11th

Colombian Computing Conference (CCC), pages 1–8.

IEEE.

Jim

enez, C., Rusu, C., Roncagliolo, S., Inostroza, R., and

Rusu, V. (2012). Evaluating a methodology to estab-

lish usability heuristics. In 2012 31st International

Conference of the Chilean Computer Science Society,

pages 51–59. IEEE.

Kuparinen, L., Silvennoinen, J., and Isom

aki, H. (2013). In-

troducing usability heuristics for mobile map applica-

tions. In International Cartographic Conference. In-

ternational Cartographic Association.

Marquez, J. O., Meirelles, P., and da Silva, T. S. (2021a). In-

teractive web maps: Usability heuristics proposal. In

Proceedings of the ICA, volume 4. Copernicus GmbH.

Marquez, J. O., Meirelles, P., and da Silva, T. S. (2021b).

Towards usability heuristics for interactive web maps.

In Proceedings of the XX Brazilian Symposium on Hu-

man Factors in Computing Systems, pages 1–7.

Nasr, E., Alsaggaf, W., and Sinnari, D. (2023). Developing

usability guidelines for mhealth applications (ugmha).

Multimodal Technologies and Interaction, 7(3):26.

Nielsen, J. (1993). Usability Engineering. Academic Press.

Nielsen, J. (1994). Usability inspection methods. In Confer-

ence companion on Human factors in computing sys-

tems, pages 413–414.

Nielsen, J. and Molich, R. (1990). Heuristic evaluation of

user interfaces. In Proceedings of the SIGCHI confer-

ence on Human factors in computing systems, pages

249–256.

Qui

nones, D. and Rusu, C. (2019). Applying a methodol-

ogy to develop user experience heuristics. Computer

Standards & Interfaces, 66:103345.

Roth, R. E., C¸

oltekin, A., Delazari, L., Filho, H. F., Grif-

ﬁn, A., Hall, A., Korpi, J., Lokka, I., Mendonc¸a, A.,

Ooms, K., et al. (2017). User studies in cartogra-

phy: opportunities for empirical research on interac-

tive maps and visualizations. International Journal of

Cartography, 3(sup1):61–89.

Rusu, C., Mu

noz, R., Roncagliolo, S., Rudloff, S., Rusu, V.,

and Figueroa, A. (2011). Usability heuristics for vir-

tual worlds. In Proceedings of the Third International

Conference on Advances in Future Internet, ser, pages

16–19.

Solano, A., Rusu, C., Collazos, C., Roncagliolo, S., Arcin-

iegas, J. L., and Rusu, V. (2011). Usability heuristics

for interactive digital television. AFIN, pages 21–27.

Vi, S., da Silva, T. S., and Maurer, F. (2019). User ex-

perience guidelines for designing hmd extended re-

ality applications. In Human-Computer Interaction–

INTERACT 2019: 17th IFIP TC 13 International

Conference, Paphos, Cyprus, September 2–6, 2019,

Proceedings, Part IV 17, pages 319–341. Springer.

Victorelli, E. Z. and Reis, J. C. D. (2020). Human-data in-

teraction design guidelines for visualization systems.

In Proceedings of the 19th Brazilian Symposium on

Human Factors in Computing Systems, pages 1–10.

Validation and Reﬁnement of Usability Heuristics for Interactive Web Maps

409