AN APPROACH TO GUIDELINE INSPECTION

OF WEB PORTALS

Andrina Granić, Ivica Mitrović* and Nikola Marangunić

Faculty of Natural Sciences, Mathematics and Education, University of Split, Nikole Tesle 12, Split, Croatia

* Arts Academy, University of Split, Glagoljaska bb, Split, Croatia

Keywords: Usability evaluation, guideline inspection, evaluation form, broad-reach web portals.

Abstract: Overall research is aiming to design discount evaluation methodology for web portal assessment. A number

of problems was identified during testing user tasks in scenario-based usability testing while others were

detected through tasks mentally simulated by HCI experts using an inspection method. This paper reports on

the experience regarding the latter one. Designed inspection has proved very promising, although obtained

comprehensive quantitative and qualitative data revealed the need for some improvements. Revision of the

evaluation form along with subsequent assessment with adequate expert sample is needed.

1 INTRODUCTION

Web portal, generally considered as a single point of

access to information, resources and services on a

wide range of topics (Waloszek, 2001), is typically

based on the more advanced technologies that go

beyond simple interface of the just information

based standard web page. Broad-reach portals, also

called "general" or "generic" portals, bring together

services such as search engines, e-mail, news,

forums, event guides, maps, on-line shopping, travel

information and the like. Accordingly, information

presented in every page can be delivered to a

number of users having different needs, motivations

and goals which portal design has to reflect.

Market research findings related to Croatian web

sphere undertaken in the last few years report that

broad-reach portals are the most visited web sites

(GFK Croatia, 2006; GemiusAudience, 2007). This

is the basic distinction from countries with high

level of Internet literacy, where there are much more

specialized web portals seen as gateways to the

variety of web information related to the specific

context (Tatnall, 2005).

When considering web portal usability, it should

be noted that current research on usability evaluation

is mostly concerned with focused, domain specific

portals, e.g. (Boye, 2006; Carstens and Patterson,

2005; Brantley et al., 2006). This is the result of

global trend of portal specialization when is quite

difficult to find any studies related to broad-reach

web portal assessments.

The objective of the overall study is the design

of "discount evaluation" approach to web portal

usability assessment. We conducted the experiment

in order to evaluate how easy and efficient the most

visited portals are. The study is placed in Croatian

web sphere where broad-reach web portals, as the

most visited sites, are much more popular than the

specialized ones. Such context seems appropriate for

this research, since it implies end users and

designers familiarity. We expected to find different

sets of problems because a wide variation in tasks

and applied diverse usability evaluation methods,

both empirical and analytic. A number of problems

was identified through testing user tasks in scenario-

based usability testing, which comprised a number

of test methods. Other problems were discovered

throughout tasks mentally simulated by experts from

the field of human-computer interaction (HCI) using

an inspection method.

Strong point in such approach is the chance to

supplement results from the guideline-based

inspection and the user-based one, enhanced with

feedback on users' pleasantness while working with

the portal. In this paper we report on and discuss the

results obtained through guideline-based inspection.

401

Mitrovi

c I., Grani

c A. and Maranguni

c N. (2008).

AN APPROACH TO GUIDELINE INSPECTION OF WEB PORTALS.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - HCI, pages 401-404

DOI: 10.5220/0001719904010404

 SciTePress

2 GUIDELINE INSPECTION

We conducted a controlled experiment which

included less strict heuristic evaluation (Nielsen,

1994), i.e. guideline inspection, supplemented with

scenario-guided user evaluations involving task-

based end user testing, usability satisfaction

questionnaire and semi-structured interview, all used

to collect both quantitative data and qualitative

"remarks", cf. (Shackel, 1991). In the following we

describe the guideline inspection along with the

acquired results and discussion of findings.

Scenario-guided user evaluation along with the

achieved results is addressed in (Granić et al., 2008).

2.1 Experts, Instruments and Measures

Particular procedure was conducted with a group of

ten "instant" specialists from the HCI field. With the

intention of overcoming the problem of not having

enough usability experts who could be involved in

the web portal evaluation process, we had the

guideline inspection performed by "instant experts"

(Wright and Monk, 1991). Those were web design

practitioners who learnt principles of good user-

centered design and provided usability expert

assessment of the selected portals.

Evaluation form consisting of a set of

principles/guidelines augmented with auxiliary

guidelines, as additional explanations related to web

portals, was prepared. Individual expert’s marks and

comments concerning the assessed portals were

collected. The score for every portal was calculated

as an average mark on a seven-point Likert scale.

Additional observations concerning the inspection

procedure could be provided as well. Four broad-

reach web portals were included in the study: Index

portal (www.index.hr), Net portal (www.net.hr), Vip

portal (www.vip.hr) and T-Portal (www.tportal.hr).

Those portals were selected as the most visited

whilst also the earliest broad-reach web portals.

2.2 Assessment Procedure

Document containing detailed instructions and

prepared evaluation form was sent to chosen experts.

Aiming to discover possible problems in the

interface design, they had to mentally simulate the

tasks to be performed on portals, mark and comment

on in the evaluation form, following the instructions

and the provided guidelines along with the auxiliary

ones. Consequently, in order to supply all necessary

information, the evaluation form had to be very

detailed and self-explanatory. Nielsen’s usability

heuristics as a set of ten key principles (Nielsen,

1994) is explained and adjusted to portal usage.

Besides, as additional explanation to the guideline, a

series of auxiliary ones concerning portal design was

also provided, cf. (Wood, 2004; MIT, 2004).

Experts had (i) to specify a level of their

conformity with a principle/guideline and related set

of auxiliary guidelines on a seven-point Likert scale

and (ii) to provide a comment to justify assigned

mark since they were encouraged to offer additional

notes related to advantages and disadvantages of

assessed portals. Furthermore, observations and

remarks concerning the overall guideline assessment

procedure were more then welcome.

2.3 Results

Experimental results and findings acquired through

guideline inspection are addressed in what follows.

Arithmetic means of marks from a seven-point

Likert scale provided by ten "instant experts"

according to ten usability guidelines show that

highest mark was given to Vip portal (mean = 5.38),

followed by Net portal (mean = 4.85), T-Portal

portal (mean = 4.64) and Index portal (mean = 4.01).

Overall achieved results could be further related

to the key experts’ comments from the form. For

Vip portal experts emphasized well-adjusted,

consistent layout, simple navigation and feeling of

control. Specialists criticized portal’s slow respond,

errors while loading and awkward forum. Net portal

is characterized as coherent with good quality of

information structure, but with poor and old fashion

visual appearance. According to experts, its main

limitations are too long initial page, unclear names

of categories, shallow navigation structure and

absence of contact information. Identified problems

in T-Portal portal are related to employed diverse

types of navigation, too long initial page, absence of

contact information, help and FAQ. Lack of

consistency and aggressive "visual noise" were the

main reasons why Index portal got severely bad

marks. Recognized problems include ambiguous

home page, lack of consistency concerning names,

categories and navigational graphics, navigation

overload, absence of help and FAQ.

The evaluation form analysis included the

assessment of adapted guidelines themselves and

judgment of the quality of experts' evaluation.

Qualitative analysis criteria were expressed in terms

of the mark span along with the significance and

usefulness of comments (see columns Mark Span

and Info in Table 1). The guidelines were examined

through expert's comments and observations,

ICEIS 2008 - International Conference on Enterprise Information Systems

402

assigning low (L), medium (M) and high (H) values

according to quantity and level of details of provided

comments (Info column). The range of marks

expresses the lowest and highest ones (Mark span

column). Quality of experts' evaluation was assessed

too. It comprised inspection of specialist's answers

to the guideline compliance related to the four

assessed portals. Similar information quality criteria

were used while analyzing experts' work: number,

percentage and quality of provided comments as

well as number of additional observations.

3 DISCUSSION

In the following the comprehensive analysis of data

obtained through ten evaluation forms is presented.

3.1 Guidelines Issues

The quality of information achieved through

individual expert’s comments was categorized as

low, medium and high. E.g. remarks like "there is no

mistake" or "not good at all" represent comments of

low information quality. Conversely, detailed ones

which list specific observations related to page

layout, fonts, navigation/links and graphics, are

classified in medium and high information quality

category.

Analyses of the marks and comments indicate

that some provide extremely poor information.

Those guidelines have wide mark span, which could

imply limitations in their understanding and/or their

vague formulation. Besides, those guidelines also

offer minimal amount of comments because they do

not stir/provoke experts in sufficient portal analysis.

Moreover, any additional information regarding

problems and possible solutions in portal design is

not offered. Examples of such guidelines are no. 1,

no. 5 and no. 9. Conversely, narrow mark span for

guidelines no. 2, no. 3 and no. 7 points toward their

clarity and unambiguity, also indicating high degree

of experts’ accordance in their valuation. Moreover,

acquired comments provided new and detailed

information about particular design problems. Our

results suggest that a good guideline is the one (i)

characterized with a narrow mark span and (ii)

"provoking" good quality comments and criticism

which detects interface design problems offering

solutions at the same time. We assume that obtained

quite low span of some guidelines implies their good

quality, despite the non-uniform group of specialists.

Table 1: Guideline analysis.

Portals

Index

portal

Vip

portal

Net

portal

T-portal

portal

Guidelines

Mark

span

Info Mark

span

Info Mark

span

Info Mark

span

Info

1. Portal is actively informing user about its'

processes (information about what is going on

is always present).

4 – 6 L 2 – 6 L 3 – 6 L 2 – 6 L

2. Concept of portal is well adjusted to the user

context.

5 – 6 M 5 – 7 M 5 – 6 L 2 – 6 M

3. While working with portal users have feeling

of control, safety and navigation freedom.

3 – 6 H 4 – 7 M 2 – 6 H 2 – 6 H

4. Portal respects media standards and usual

practice/usage/routine.

1 – 4 H 5 – 7 M 4 – 6 M 2 – 7 M

5. Portal prevents possible user errors. 2 – 6 L 3 – 7 L 3 – 7 L 2 – 7 L

6. User is intuitively getting information on

portal, i.e. user doesn't have to remember

information path but recognize it.

4 – 5 M 4 – 7 L 3 – 6 M 2 – 6 L

7. Portal is adjusted for efficient use by novice

users as well as by experts.

3 – 5 H 4 – 6 M 3 – 6 H 2 – 6 M

8. Portals' design is clear, understandable and

transparent, i.e. most needed information are

at the same time most visible.

2 – 5 M 5 – 7 M 4 – 6 M 2 – 6 L

9. Portal enables user recognize and help

recover from errors.

2 – 6 L 3 – 7 L 2 – 7 L 3 – 7 L

10. Portal offers help while working on it. 2 – 5 M 3 – 6 M 3 – 6 M 2 – 7 M

AN APPROACH TO GUIDELINE INSPECTION OF WEB PORTALS

403

3.2 Instant Expert Issues

Further analysis of evaluation forms provides insight

into instant experts sample selection. Apart from

marks, three specialists did not provide any

supplementary comment or observation concerning

the overall assessment procedure. Two experts

supported their marks with very poor explanations

and quality of acquired information didn’t help to

get insight into design strengths and weaknesses.

On the other hand, five experts provided

evaluation forms with marks and detailed comments

of every guideline for each portal. Additionally they

offered remarks and suggestions for an improvement

of the evaluation process in general and individual

guidelines in particular. Such considerable

distinction in acquired information quantity and

quality suggests quite heterogeneous group of

experts in their HCI knowledge and usability

expertise. This problem was very difficult to avoid

due to limited number of HCI experts in Croatia on

the one hand and high costs of foreign experts’

engagement on the other.

4 CONCLUSIONS

The objective of the research is the design of

"discount evaluation approach" to web portal

assessment. Aiming to achieve this goal, design of

most visited Croatian broad-reach portals was

assessed both through usability inspection method

and a number of usability test methods.

We found that designed evaluation form used in

guideline inspection needs significant revision.

Some of adapted Nielsen’s principles showed poor

applicability in web portal context. The derived

guidelines did not provide any information which is

useful for improving portals’ usability. For that

reason, a number of guidelines should be more

comprehensible, auxiliary guidelines revised and

redundant ones excluded. A new set of guidelines is

needed, the one which is not so strictly based on

Nielsen’s heuristics.

Significant difference in acquired experts’

information suggests the non-homogeneity of

"instant experts" group concerning their HCI

expertise. Such problem was hard to prevent due to

inadequate number of resident HCI experts as well

as high costs of possible foreign specialists’

engagement.

The guideline-based evaluation, even though

showing respectable potential, raised a couple of

concerns. In order to upgrade the applied usability

inspection method (i) the "instant experts" selection

and the evaluation form issues should be revised

according to the quantitative and qualitative analysis

of the obtained results and (ii) the redesigned

method should be applied in an assessment of more

specialized Croatian web portals.

ACKNOWLEDGEMENTS

This paper describes the results of research being

carried out within the project 177-0361994-1998

Usability and Adaptivity of Interfaces for Intelligent

Authoring Shells funded by MZOŠ of the Republic

of Croatia. The experiment was conducted at Dept.

of Visual Communication Design, Arts Academy.

REFERENCES

Boye, J., 2006. Improving portal usability. CMS Watch.

http://www.cmswatch.com/Feature/151-Portal-

Usability.

Brantley, S., Armstrong, A., Lewis, K.M., 2006. Usability

Testing of a Customizable Library Web Portal.

College & Research Libraries 67, 2.

Carstens, D.S., Patterson, P., 2005. Usability Study of

travel websites. Journal of Usability Studies, 1, 1.

GemiusAudience, 2007.

http://www.cati.hr/GemiusAudience-ozujak.pdf.

GFK Croatia, 2006. ttp://www.gfk.hr/press1/internet.htm.

Granić, A., Mitrović, I., Marangunić, N., 2008. Experience

with Usability Testing of Web Portals. 4

Int. Conf.

on Web Information Systems and Technologies. 4-7

May, 2008, Madeira, Portugal (accepted for

presentation)

MIT Usability Guidelines, 2004.

http://web.mit.edu/is/usability/selected.guidelines.pdf.

Nielsen, J., 1994. Heuristic Evaluation. In Nielsen, J.,

Mack, R.L. (Eds.): Usability Inspection Methods. John

Wiley and Sons Inc, New York.

Shackel, B., 1991. Usability – context, framework, design

and evaluation. In Shackel, B., Richardson, S. (eds.):

Human Factors for Informatics Usability, Cambridge

University Press, Cambridge, 21-38.

Tatnall, A. (Ed.), 2005. Web portals: the new gateways to

Internet information and services. Idea Group

Publishing, Hershey.

Waloszek, G., 2001. Portal Usability – Is There Such A

Thing?. SAP Design Guild, Edition 3.

http://www.sapdesignguild.org/editions/edition3/

overview_edition3.asp.

Wood, J., 2004. Usability Heuristics Explained, iQ

Content. http://www.iqcontent.com/publications/

features/article_32.

Wright, P., Monk, A., 1991. A cost-effective evaluation

method for use by designers. International Journal of

Man-Machine Studies 35, 6, 891-912.

ICEIS 2008 - International Conference on Enterprise Information Systems

404