TOWARDS USABILITY IMPROVEMENT OF SEMANTIC WEB
APPLICATIONS
Llúcia Masip, Marta Oliva, Roberto García and Toni Granollers
Department of Computer Science and Industrial Engineering, University of Lleida, Lleida, Spain
Keywords: Usability, Semantic Web, Heuristic Evaluation.
Abstract: In recent years, a new revision of the Web is being proposed. It goes beyond Web 2.0 so it is called the Web
3.0.We concentrate on just one of them, the one called the Semantic Web. This proposal has been
technologically developed and improved but its lack of taking off among end-users seems due to problems
in the end-user interactive experience. In this paper, we explore this aspect because it is as important to
improve Web 2.0 applications as improving Web 3.0 interfaces and it is as important to develop technology
as emphasizing its new user interfaces. Thus, we present a usability evaluation carried out through the
heuristic evaluation in eleven semantic web applications to detect common misconducts. In addition, we
also assess if we can use the same heuristics for a semantic web applications and for a common Web
applications.
1 INTRODUCTION
Semantic Web (SW) and Web 3.0 are emerging
terms that appeared to go beyond current Web, being
currently accepted that the former is a subset of the
latter. Even accepting that these proposals can
provide a “better Web”, it is not obvious that they
will improve the end-user interactive experience
when using Web 2.0 based interfaces.
However, Web 3.0 should not be viewed as a
replacement of Web 2.0, it extends (or enhances)
Web 2.0 through new possibilities. These provide
new tasks that present the same basic goal than other
tasks include in a common interface, but these tasks
increase user experience making actions such as
sharing or searching content easier.
We agree with James Kalbach blog’s words
(Kalbach, 2010): “The success of next-generation
information systems depends much more on human
factors than on more sophisticated technologies”. It
means the paradigm used to design a user interface
should not be realized by final users; paradigms
must be transparent layers to them.
Apart from that and in the same way as other
kind of user interfaces, we have to consider user
experience factors in interface design to improve it
and achieve a new interface that makes it easier to
have a positive experience. And, for achieving it we
do not see other way than including end users in
interface design, following the User Center Design
(UCD) principles (Abras et al., 2004).
As UCD practitioners, as a rule, we consider end
users when designing an interface such as a Website.
But, some questions such as “what happens with a
semantic web application?” come up to our minds.
According to Peter Morville in (Morville, 2005):
“As interface stands on the shoulders of
infrastructure, tomorrow's user experience will rest
on the foundation of today's Semantic Web
technologies". We have to make efforts to improve
the visible layer of the interfaces. And we have to
achieve that end users can carry out their tasks in a
usable way to improve their interactive experience.
Nowadays, SW is mainly used in the backend
part of the system (Cyganiak and Jentzch, 2010). We
consider that there is much work to do for improving
its positive experience of use and bringing it also to
the frontend. We completely agree with the creator
of SW, Tim Berners-Lee, when he said (Berners-
Lee, 2007): “We do not yet have Semantic Web
technology available which is that easily usable by
grandparents and children. That is true."
Many resources exist and we use them to design
SW applications but, why do we not add to these
resources aspects (such as usability) that we consider
necessary for improving user experience? Ora
Lassila adds: "After 10+ years of work into various
aspects
of the Semantic Web, I am now fully
361
Masip L., Oliva M., García R. and Granollers T..
TOWARDS USABILITY IMPROVEMENT OF SEMANTIC WEB APPLICATIONS.
DOI: 10.5220/0003905803610366
In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST-2012), pages 361-366
ISBN: 978-989-8565-08-2
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
convinced that most of the remaining challenges to
realize the Semantic Web vision have nothing to do
with the underlying technologies. Instead, it all
comes down to user interfaces and usability.
(Lassila, 2007)"
Thus, we did research on the user interface part
of the SW and in this paper we present a list of
common misconducts that we found. For this, we
carried out an experiment analyzing 11 interfaces
that use SW technology.
In the next section we present some works
related to our article. Then, a SW evaluation is
showed in section 2. Section 3 shows the obtained
results. And finally, discussion, conclusions and
future works are presented.
1.1 Related Work
The general goal of this paper is to include usability,
as one of the most important factors used in user
experience, in a SW design such as in (García et al,
2010).
User experience is a relatively new concept
which according to our research approach includes
different facets (Masip et al, 2011), being usability
one of these facets. To now, for a long time,
scientific community used only usability features to
improve the quality of interactive systems.
Nowadays, this tendency is changing; experts
see the interactive experience as a whole and include
more facets than usability to achieve a positive user
experience. As a first step in SW application, we
start to improve the quality of this type of interfaces
including usability features in its design. Because we
consider if SW designers apply specific usability
aspects in their designs, user experience will be
better.
There are papers in literature concerning the
usability of SW applications but, usually, these are
focused on few features of semantic applications
such as natural language (Cimiano et al, 2008)
(Dittenbach et al, 2003) or search engines (Reichert
et al, 2005) (Duke et al, 2007).
Furthermore, we found works concerning
usability problems in SW applications. For instance,
in (Roy et al, 2010) the authors justify that if we use
ontology, user experience will improve. Another
work is (Jamenson, 2006), where the author shows
usability aspects about some tasks. But, works do
not present details. We try to do so in the following
sections.
2 SEMANTIC WEB EVALUATION
We consider that the first step to improve user
experience of semantic applications is including one
of the most used factors in user experience, the
usability (Dix et al, 2004) (Nielsen et al, 1994).
So, in this paper we used heuristic evaluation to
check a set of chosen Websites and show their
common usability problems. We use heuristic
evaluation because is fast, cheap and one of the
oldest methods to evaluate the usability of the
interactive systems. In the following sections, we
present the evaluated Websites, the method used and
the process followed to evaluate these interfaces.
2.1 Set of Evaluated Websites
Sometimes, it is obvious for a SW expert that one
Website is designed with semantic technology, but
other times, this semantic technology goes unnoticed
and it is very difficult to detect it. However, the
technology
used to design a Website should be
Table 1: Websites for specialized users.
URL User profile
http://iserve.kmi.open.
ac.uk/browser.html
iServe is the place on the
Websites where linked
data meets services. So, it
is a platform for
publishing Semantic Web
Services as linked data,
no matter their original
format. (from:
http://iserve.kmi.open.ac.
uk/wiki/index.php/Home)
.
http://lod.openlinksw.
com/
Everybody who wants to
add this engine in search
bar of an OpenSearch -
capable browser. But
those who use this engine
will be an expert finder
of information.
http://www.faviki.co
m/pages/welcome/
Everybody who wants to
keep their own tags and
connect them in common,
universal concepts from
the world's largest
collection of
knowledge.(from:
http://www.faviki.com)
http://dblp.rkbexplore
r.com
Semantic web expert who
is interested in RDF
linked data and co-
reference information.
http://sig.ma/
Everybody who wants a
Web of Data browser
Website, an
embeddable/linkable
widget and a semantic
API.
WEBIST2012-8thInternationalConferenceonWebInformationSystemsandTechnologies
362
transparent to end users. So, a good Website design
has been done if we (as SW experts) cannot detect
semantic technology in it.
Therefore, having usability improving of SW
application as a goal, we chose eleven sites that we
knew that they use semantic technology in their
implementation. And, while we were choosing these
Websites, we detected that Websites can be (and
should be) classified into two different groups:
Websites for end-users (general audience) and
Websites for specialized users.
Websites for end-users are Websites where
everybody can visit and they can understand its
content. So, these Websites are for users who should
not have any knowledge of semantic technology to
be able to use them.
On other hand, Websites for specialized users are
Websites where, obviously, everybody can visit, but
only expert users who know the content meaning
will understand the information provided because
they have some knowledge of semantic technology.
In the following two tables, every Website of
each group and a brief description about a specific
user profile who can use these webs are shown.
Table 2: Websites for end-users.
URL User profile
http://news.bbc.co.uk/sp
ort2/hi/football/world_cu
p_2010/default.stm
Everybody who likes football
and especially, the World Cup.
http://www.uptake.com
Everybody who wants to learn
about new places or something
special in a specific city
around the world.
http://www.viewchange.
org
Everybody who wants to
browse, watch or share videos
about real people and progress
in global development.
http://www.zemanta.com
Everybody who frequently
uses web word processors and
need content and picture
suggestions related to
composition.
http://www.freebase.com
Everybody who wants to
browse, see or share data in an
open way.
http://www.data.gov/
Everybody who wants to know
much information about the
Government data stored in
many locations.
2.2 Usability Evaluation
SW distinguished itself by showing specific tasks or
information that in a usual Website we normally do
not find it. However, knowing that one of our aims
is to assess the evaluation methodology itself, we
will attempt evaluating these interfaces using the
same method that we use in a “traditional Website”.
It will provide us with useful information to consider
if the method (or part of it) needs to be adapted. So,
we decided to use one of the most used methods to
evaluate usability: heuristic evaluation (Dix et al,
2004) (Nielsen et al, 1994). It is an inspection
method that allows usability experts to get
improvements faster and cheaper than other
evaluation methodology such as user test. Therefore,
heuristic evaluation also helps us to get the list of
misconducts.
In previous works, we got 16 usability categories
defined by at least 3 authors (Masip et al, 2010).
Now, these categories are used to evaluate interfaces
that apply semantic technology to their design, and
we explore if our set of heuristics is complete
enough to detect all problems for this type of
interfaces. Bearing in mind that semantic technology
must be transparent for end users.
2.3 Evaluation Process
We followed the generic process that Nielsen
presented (Nielsen et al, 1994). The steps to carry out
the heuristic evaluations are:
First review: one evaluator navigated through
each interface to make first contact with it.
She was expert in heuristic evaluation but she
was non-expert in SW applications.
Heuristics scores: every heuristic was scored
and some specific observations in each
heuristic were written. She used the same set
of heuristics for all Websites. And she
scheduled only 3 usability evaluations per day,
avoiding an extreme memory load and the
comparisons between systems.
Finally, we could summarize the quantitative
and/or qualitative results.
3 RESULTS
Our goals were not to present specific usability
problems detected in each Website according to
scores used. Our main goals were: First, detecting
common mistakes in SW applications. Second,
assessing if the heuristics used are complete enough
for this type of interfaces.
3.1 Misconducts
As we mentioned, we present every mistake that we
found at least in three evaluated interfaces:
TOWARDSUSABILITYIMPROVEMENTOFSEMANTICWEBAPPLICATIONS
363
Interfaces do not include a visible navigate
resource (next and before), so you can only
use menu options presented in the navigator.
If we usually use cursor, in these interfaces
cursor movement is not consistent.
Websites do not show sitemap, so, we
cannot see a general view of the Website.
Computer experts appreciate shortcuts
because they make easier and more quickly
some tasks. In the analysed Websites there
are not clear shortcuts to the main tasks.
Many Websites are difficult to use; even so,
they do not show any help section or
documentation. What can you do in this
interface it is not so clear.
There are jargon words about Web content
or ontology vocabulary. This ontology
vocabulary is too specific for end-users.
And in the same way than the point above, it
shows not familiar vocabulary. In fact, it
causes much confusion to end-users.
We know that in semantic web area URLs
are very important but even so, these URLs
are not understandable to users.
In semantic applications, information is
related itself but this relation is not visible in
end user layers. Each menu tag is not visibly
associated with upper menu tag.
Due to new task presented in semantic web
applications, some of them are difficult to
learn and carry out.
We consider that in Web application is very
important to show update dates because
Website is changeable by nature. But,
actualization date does not appear.
In reference with the last point, many dates
that appear in these Websites do not follow
international format, they do not include p.m
or a.m abbreviations and they do not show
the month in letters to avoid confusion
between days and months.
Websites do not highlight link areas when
the cursor is on it. Taking into account that
in semantic web applications many links
appear, user does not know where cursor is.
Semantic applications display many
pictures. We realized that some of these
pictures do not have the label tag.
Contact information does not appear clearly.
Website can be visited by many people that
live in different countries with different
cultures and languages. So, including
language options is very important.
Link labels are not appropriate.
Designers use some orientation resources
such as breadcrumbs for locating users in
the Website, but the analyzed Websites lack
this type of orientation information.
Interfaces should be intuitive but in the
Websites analyzed the user were forced to
remember previous experiences to be able to
carry out a task again.
Reducing user memory load is essential to
show information in a clear way, without so
much information. The Websites evaluated
present too much information. For instance,
if we search “Leo Messi” in sig.ma Web
page, we see an enormous scroll with a lot
of information.
We consider that in Websites we should
show enough information to carry out tasks
without problems.
According to other misconducts such as the
vocabulary used, the main goals of each
interface are not so clear. But this mistake is
a consequence of other cited problems.
If a search does not have results, system
does not warn users.
We should know the meaning of the tables
with only catching a glimpse of it. When
tables are shown in interfaces, their columns
or rows titles are not representative and
understandable.
Another consequence about understandable
links is the opening of new windows without
warning users. And sometimes, these links
or pages are “Not found” pages.
It is important that the interface uses the
same aesthetic design in the site. However,
Websites display different aesthetic design,
and sometimes we (as evaluators) thought
we were in a different Website.
To sum up, the general impression is that
navigation systems are poor.
3.2 Quantitative Results
Quantitative results will be analysed distinguishing
between Websites for end-users and Websites for
specialized users. There are many repeated
problems. Nevertheless some different aspects were
identified, the most important being that Websites
for specialized users present more usability
problems. It is due to Web designers not giving
importance to specific features that are essential in a
Website for end-users. In next table we show that in
all Websites 63 different problems appeared. From
these, 48 are common in both types of Websites but
the remaining fifteen are separated as follows:
WEBIST2012-8thInternationalConferenceonWebInformationSystemsandTechnologies
364
Websites for end-users have 4 problems that
Websites for specialized users do not suffer, and
Websites for specialized users have 11 problems that
do not appear in Websites for end-users.
Table 3: Classification of problems.
Total of
problems
Equal
problems
Different problems
63 48
Websites for
end-users
Websites for
specialized
users
4 11
In usability evaluation section we cited a set of 16
usability categories that we used to carry out all
heuristic evaluations (Masip et al, 2010). Now, we
present a table with the amount of misconducts that
we detected in each category. We detected 63
different problems in all Websites, which can be
summarised into 26 misconducts detected at least in
three Websites. Note that we do not show all
categories because we did not find misconducts in
all of them. So, we do not include categories without
problems in the next table:
Table 4: Quantity of problems.
Category Number of problems
Navigation 4
Memory load 4
Dialogs 4
Internationalization 3
Content 3
Consistency 2
Shortcuts 1
Help 1
Search 1
Flexibility 1
Easy use 1
Feedback 1
4 DISCUSSION
According to the evaluation, the weakest usability
categories in this type of Websites are navigation,
memory load, dialogs, internationalization, content,
consistency, shortcuts, help, search, flexibility, easy
use and feedback. Problems found in navigation
category are related with menus structure. It is worth
highlighting that some interfaces analyzed are
browsers, so, menu is poor or null. The same
happens when analyzing where the information is
located. It is clear that if the ontology appears, the
Website might present a complex structure so, it is
impossible to use the traditional breadcrumbs. But it
is also clear that ontology’s might have hierarchy of
information. So designers should try to create
breadcrumbs or something similar to locate users in
the interface using the upper classes and show
breadcrumbs.
Regarding to dialog, it uses more ontology
expressions in Websites for specialized users than in
Websites for end-users. And these expressions are
closely linked to ontology jargon. It often causes that
vocabulary used is not intuitive for common users
or/and users expert with knowledge included in the
ontology. Heuristics about memory load, apart from
their links to the abusive use of specific language,
refer to the amount of information that can be found
in some pages. There is an overload of information
and links whether user is a novel or expert user.
About internationalization, we highlight that it is
very important to show dates in each part of content
because it is important to know the validity of the
content and their last updating. And we cannot
forget that if we use date and times, we will show
the time zone to locate our area in a more specific
way and they have to use international format. When
the design suddenly changes, the most important
problem in consistency category appears. Finally,
Websites search results are presented in an unclear
way and often they are impossible to understand.
But, in the same way such other problems, this type
of problems also appears in other type of Websites.
So, it is not a typical problem of SW applications but
it is important to highlight its appearance.
According to our Websites classification
(Websites for end-users and Websites for specialized
users), we detected that in Websites for end-users,
interface design is more accurate and designers spent
more efforts to achieve a Website aesthetically better
than design of Websites for specialized users. So,
Websites for specialized users present more usability
problems than other type of Websites. Furthermore,
in table 4, we note that categories in which interface
designers should do more efforts to improve them
are navigation, memory load, dialogs,
internationalization, content and consistency,
respectively. However, the worst categories are
navigation, memory load and dialogs. So, these
should be the first features that designers should
improve immediately.
5 CONCLUSIONS AND FUTURE
WORK
In this paper, a usability evaluation of Websites
those use SW technology in their design has been
TOWARDSUSABILITYIMPROVEMENTOFSEMANTICWEBAPPLICATIONS
365
presented. We proved that we can use the same
usability evaluation method in semantic application
than in another common type of application. In
addition, we proved that the set of used heuristics is
applicable in SW application and in another type of
interactive system that not use this kind of
technology. So, extending our set of heuristics to
evaluate this type of interfaces is not necessary yet
because it covers all usability features for SW
applications. Moreover, another and important
reason is that technologies used to design some
application should try to go unnoticed by end users.
But if in few years new functionalities will appear,
we should extend usability categories to cover all
features in SW applications. However, we will do
the same in another type of application. In fact, it is
technology evolution. Even so, we realized that it is
very important to design for users and include user
experience features in all interfaces design. Then,
considering that usability is one of the oldest factors
that scientific community uses to improve the user
experience of product, we included this factor in SW
application as the first one to start improving the
user experience of this type of interfaces.
We carried out a usability evaluation in eleven
semantic Websites and we concluded that the main
problems are in navigation, dialogs and memory
load categories. So, semantic application designers
should make more efforts to achieve a SW
application that causes end user a positive
experience. We hope that this article will help
semantic application designers to realize that when
they include some semantic tasks for users, these
tasks should be easy and intuitive. Our future work
will be based on extending user experience
evaluation to other factors apart from usability.
ACKNOWLEDGEMENTS
The work has been supported by Spanish Ministry of
Science and Innovation through the Open Platform
for Multichannel Content Distribution Management
(OMediaDis) research project (TIN2008-06228) and
by Universitat de Lleida for pre-doctoral fellowship
to Llúcia Masip.
REFERENCES
Abras, C., Maloney-Krichmar, D., & J. Preece., (2004)
User-Centered Design. In Bainbridge, W.
Encyclopedia of Human-Computer Interaction.
Publications, (2004).Berners-Lee, T., 2007. The future of
the Web as seen by its creator. Interview with Peter
Moon, IDG Now, July 9.
Cimiano, P., Haase, P., Heizmann, J., Mantel, M., 2007.
ORAKEL: A Portable Natural Language Interface to
Knowledge Bases, Technical Report 1439, Institute
AIFB, University of Karlsruhe.
Cyganiak, R., Jentzsch, A., 2010. The Linking Open Data
cloud diagram, http://richard.cyganiak.de/2007/10/lod/
Dittenbach, M., Merkl, D., Berger, H., 2003. A Natural
language query interface for tourism information, in:
10th International Conference on Information
Technologies in Tourism (ENTER), Helsinki, Finland.
Dix, et al., 2004. Human-Computer Interaction. Prentice
Hall, ISBN: 0130-461091.
Duke, T. Glover, J. Davies, 2007. Squirrel: an advanced
semantic search and browse facility, in: 4th European
Semantic Web Conference, Innsbruck, A.
Jamenson, A., 2006. Usability and the Semantic Web.
Invited talk at the Third European Semantic Web
Conference, Budva, Montenegro.
James Kalbach’s blog: http://experiencinginformation.
wordpress.com/2010/05/05/the-problem-with-the-sem
antic-web-usability (27-Jan-2011)
Lassila, O., 2007. Semantic Web Soul Searching. Wilbur-
and-O blog, March 19.
Masip, L., Oliva, M., and Granollers, T., 2010. Hacia la
semiautomatización de la evaluación heurística:
Primer paso, categorización de heurísticas, Interacción
2010, Grupo Editorial Garceta, Valencia, 7 -10 of
September, pp. 359-368.
Masip, L., Oliva, M., Granollers, T., 2011. User
experience specification through quality. Human-
Computer Interaction – INTERACT 2011. Lecture
Notes in Computer Science. Springer Berlin /
Heidelberg ISBN: 978-3-642-23767-6. pp 656-660
Vol 6949.
Morville, P., 2005. Ambient Findability. O'Reilly Media,
ISBN: 978-0-596-00765-2
Nielsen, J., Mack, J. L., 1994. Usability Inspection
Methods, John Wiley & Sons, New York. ISBN: 0-
471-01877-5.
Reichert, M., Linckels, S., Meinel, C., Engel, T., 2005.
Student’s perception of a semantic search engine, in:
IADIS Cognition and Exploratory Learning in Digital
Age (CELDA 2005), Porto, Portugal.
Roy, N., Brooks, K., Julien, C., 2010. Usability of
Semantic Web for Enhancing Digital Living
Experience. Consumer Communications and
Networking Conference, pp.1-6.
WEBIST2012-8thInternationalConferenceonWebInformationSystemsandTechnologies
366
