The Role of Community and Social Metrics in Ontology Evaluation:

An Interview Study of Ontology Reuse

Marzieh Talebpour, Martin Sykora and Tom Jackson

School of Business and Economics, Loughborough University, Loughborough, U.K.

Keywords: Ontology Evaluation, Social Quality Metrics, Ontology Reuse.

Abstract: Finding a “good” or the “right” ontology for reuse is an ongoing challenge in the field of ontology

engineering, where the main aim is to share and reuse existing semantics. This paper reports on a qualitative

study with interviews of ontologists and knowledge engineers in different domains, ranging from

biomedical field to manufacturing industry, and investigates the challenges they face while searching,

evaluating, and selecting an ontology for reuse. Analysis of the interviews reveals diverse sets of quality

metrics that are used when evaluating the quality of an ontology. While some of the metrics have already

been mentioned in the literature, the findings from our study identify new sets of quality metrics such as

community and social related metrics. We believe that this work represents a noteworthy contribution to the

field of ontology engineering, with the hope that the research community can further draw on these initial

findings in developing relevant quality metrics and ontology search and selection.

1 INTRODUCTION

Ontologies play a very important role in the field of

knowledge and information management by

furnishing the semantics to the semantic web

(Shadbolt et al., 2006) and are used in different

domains for various purposes. Ontologies have

many benefits, no matter in which domain they are

used. They facilitate communication and knowledge

transfer between systems, between humans, and

between humans and systems (Bürger and Simperl,

2008) by uniquely identifying the meaning of

different concepts in any domain. They can also

avoid the costs associated with new developments of

knowledge models.

Despite the significant role that ontologies play

in the semantic web, there is still little understanding

about the way they should be developed and built

(Ding and Foo, 2002). Some believe that the cost of

building and maintaining ontologies in certain

domains can outweigh the potential benefits gained

by using them (Shadbolt et al., 2006). To deal with

this concern, some have suggested reusing

previously built ontologies, since this will help in

achieving one of the main goals of ontology

construction, that is to share and reuse semantics

(Simperl, 2009), and will also save significant

amount of time and financial resources. Uschold et

al. (1998) believe that the future of construction of

large-scale knowledge-based systems is highly

dependent on reusing the components built by

others.

Regardless of all the advantages of reusing

ontologies and the availability of different

ontologies, ontology reuse has always been a

challenging task (Uschold et al., 1998). Methods for

building ontologies are usually blamed for lack of

reuse strategy and some argue that these

methodologies are not explicitly concerned with

ontology reuse (Annamalai and Sterling, 2003).

Others consider the first steps of ontology reuse,

which is identification and evaluation of the

knowledge sources that can be useful for the

application domain (Bontas et al., 2005), as the

hardest step in ontology reuse. Researchers not only

have to find the most appropriate ontologies for any

search query, but they should also be able to

evaluate those ontologies according to different

implicit or explicit criteria.

This study aims to address some of the

challenges that are faced in the first steps of the

general process of reusing ontologies, which is to

evaluate and then select a good ontology for reuse.

This study contributes with qualitative data and

findings to this ongoing challenge by documenting

Talebpour M., Sykora M. and Jackson T.

The Role of Community and Social Metrics in Ontology Evaluation: An Interview Study of Ontology Reuse.

DOI: 10.5220/0006589201190127

In Proceedings of the 9th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (KEOD 2017), pages 119-127

ISBN: 978-989-758-272-1

the process of selecting an ontology for reuse. It

differs from previous studies, which focused purely

on evaluating pre-selected metrics. In this study, our

focus was to qualitatively understand the process

and reasoning behind ontology selection and reuse,

with a particular interest in the under-researched

social and community aspects of ontology quality.

Interviews were used to understand how ontologists

and knowledge engineers in different domains

search for, evaluate and select an ontology for reuse.

This research asks:

1. What are the main characteristics of a

reusable ontology?

2. What are the main metrics used to evaluate

the quality of an ontology before selecting

it for reuse?

3. Do knowledge and ontology engineers

consider community related metrics e.g.

who has built the ontology, who has used

the ontology, etc. before selecting an

ontology for reuse?

2 BACKGROUND

The main goal of ontology evaluation is to asses an

ontology for the following purposes:1) to detect

faults in an ontology and to measure its correctness

(Hlomani and Stacey, 2014), 2) to evaluate its

quality and to help in the selection process (Hlomani

and Stacey, 2014), and 3) to track the process in

ontology development (Yu et al.,2009). Ontology

evaluation can also be done in different stages of

ontology development namely while building an

ontology, for checking inconsistencies in those

ontologies that were built automatically and last but

not least, while selecting an ontology for reuse

(Tartir

et al.,2010).

There are various ontology evaluation methods

and several ways of classifying them in the

literature. According to Brank

et al. (2005) ontology

evaluation can be done in four major ways:

evaluating an ontology by comparing it to a “golden

standard” 2) evaluating an ontology by comparing it

to a source of data 3) evaluating an ontology by

running it in an application as part of a system and

evaluating the resulting performance, 4) asking

human experts to evaluate an ontology against a set

of predefined quality criteria.

From all the aforementioned methods, metric-

based approaches (4) are very popular and different

researchers have attempted to introduce various

metrics that can be used to evaluate ontologies and

help in the decision making process for ontology

selection. The aim of this method, that is also called

featured-based approach, is to offer a quantitative

perspective of evaluating ontologies by gathering

data and meta-data on different aspect of the

ontology (Arpinar et al.,2006). Ontometric (Lozano-

Tello and Gómez-Pérez, 2004) for example consists

of a detailed set of 117 criteria to examine different

dimensions of ontologies namely content, language,

ontology construction methodologies, tools, and

costs. While many of the criteria in metric-based

evaluation approaches aim to measure different

components of an ontology e.g. content, structure,

coverage, etc., some of these have focused on non-

ontological and social aspects (McDaniel

et al.,

2016) of ontologies like popularity (Martínez-

Romero et al., 2014; Fernández et al., 2009; Wang et

al., 2008).

Despite the widespread use of the terms

popularity or acceptance in the literature, there is

still no consensus on the definition of these terms.

Popularity and acceptance tend to be mostly used to

refer to the number of times an ontology has been

viewed or used in a specific repository. NCBO

Ontology recommender for example, calculates the

popularity of an ontology by checking the presence

of the ontology in well-known repositories as well as

looking into the number of visits or pageviews to an

ontology in ontology repositories in a recent specific

period (Martínez-Romero et al., 2017). In the paper

by Burton-Jones et al. (2005) the authors also refer

to the term history to indicate the number of times

an ontology has been used.

The second definition of popularity is based on

applying the PageRank algorithm (Page et al.,1999)

to ontology engineering field and focuses on the

import feature of ontologies. Fernández

et al. (2009)

for example has defined the term “direct popularity”

as the number of ontologies importing a given

ontology. Wang

et al. (2008) used the same

definition to define what they call popularity, that

for them is measured by considering how much an

ontology is referenced by others. As a part of the

authority metric in Supekar

et al. (2004), authors

have mentioned a metric called citation and have

defined it as the number of occurrence of

daml:sameClassAs, rdfs:seeAlso, owl:imports in a

given ontology.

3 METHODS

Semi-structured interviews with ontologists and

knowledge engineers were conducted to investigate

the thinking behind and the processes commonly

involved in evaluating ontologies for their reuse.

Purposive sampling was used to find the experts in

the field of ontology engineering. Different sampling

strategy namely intensity sampling was applied to

find the ontologies that have been reused and then to

interview the individuals who had built or had

reused those ontologies. Moreover, homogenous

sampling was used to find different ontology related

research groups in different organisations and

universities working in different domains.

We interviewed 15 researchers with different

levels of expertise and knowledge engineering

backgrounds. As it is seen in the table 1, four out of

the fifteen interviewees had only worked in the

biomedical field, five had some biomedical

experience but had also worked in other fields such

as computer science, and the rest of the interviewees

were mostly involved in manufacturing, smart cities,

etc. The semi-structured interview protocol focused

on how each individual (i) built, (ii) searched for,

(iii) evaluated and (iv) reused ontologies. Interviews

ranged from 20 to 60 minutes, all of which were

conducted via Skype. Interviews were recorded, and

the interviewer took field notes during the interview.

Field notes and transcriptions were coded using

NVivo.

Interviews were conducted until no new

information or theme was found (Guest et al.,2006)

and the conceptual saturation was reached. The

sample size can also be justified by some of the

previous similar research on ontology evaluation for

example the survey that was conducted by Tello

(2002), which had 10 participants. Based upon the

research questions, we began by coding for 1)

building a reusable ontology, 2) characteristics of a

reusable ontology, 3) finding a reusable ontology, 4)

evaluating/trusting/selecting ontologies, and 5) the

importance of community.

4 FINDINGS

According to the interview findings, metrics for

evaluating the quality of an ontology for reuse can

be classified into the following categories:

 metrics based on the ontology components

including content, structure, coverage, etc.

 metrics related to the metadata about an

ontology such as methodology,

documentation, language, etc.

 metrics related to community, popularity,

and ontology developer team

The main focus of this paper is on the

community and social aspects of ontologies. The

following parts of this paper moves on to describe in

detail how participants in the interviews intended to

refer to the community to search for, find and

evaluate an ontology for reuse.

Table 1: Domain Expertise of Ontologists and Knowledge

Engineers Interviewed.

4.1 Community and Ontology Search

One of the fundamental objectives of the interviews

was to explore the search process for reusable

ontologies. Consequently, the question “how do you

find the ontology you want to reuse?” was asked and

while the researcher was expecting to hear about

some popular search engines in ontology

engineering domain like Swoogle, BioPortal, etc.,

literature and published papers were mentioned by

many of the interviewees as one of their main

sources of finding the ontologies they need.

Interviewee NBI4 for example, blamed his

domain for lack of good and well-established

repositories for ontologies and said that “I go to the

literature”. Another interviewee, SB3, also

Name/Code

Role / Domain, organisation,

or project

BI1

Group leader /

Bioinformatics, Gene

ontology

BI2

Researcher / BioPortal

BI3

Ontology Developer /

Bioinformatics, Gene

ontology

BI4

Researcher / Biomedical

Informatics

SB1

Ontology developer /

Industry, W3C, NHS

SB2

Researcher / BioPortal

SB3

CEO and ontology developer

/ Bioinformatics

SB4

Lecturer / Computing Science

and Biology

SB5

Research scientist / Protégé

group

NBI1

Ontologist / IBM, Smarter

Planet Project

NBI2

Professor, Manufacturing

Informatics

NBI3

Ontology engineer / Semantic

Web

NBI4

Researcher / Laboratory for

Applied Ontology

NBI5

Researcher / Smart Cities,

Geo Ontologies

NBI6

Researcher / Industrial

ontologies

emphasised the significant role of literature in the

process of searching for ontologies and mentioned

that “reading publications around the ontology” is a

very good method to help find the ontology,

especially if someone is new to the field.

Besides helping to find a reusable ontology,

some of the other interviewees stated that they use

the literature and research papers as a tool to

evaluate the quality of an ontology. Respondent

NBI4 pointed out:

If an ontology is good and is used, you find

a cite in the literature.

Being based on published research papers will

not only affect the quality of an ontology, but

according to some of the respondents, will also

affect the popularity of an ontology; BI4 for

example stated:

Popular ontologies are better ontologies,

people are just familiar with popular

ontologies so whenever you go to any

ontology related conference, you will

always have a workshop or a paper that

talks about the ontology

4.2 Community and Ontology

Evaluation

As was highlighted in section 2, various work has

looked at the quality and evaluation of ontologies,

however while some of the papers have attempted to

cover the social aspects of ontology evaluation, none

have gone further than measuring popularity,

authority, and history of ontologies and almost all of

them have neglected the other interactions in the

community that can affect the way ontologies are

evaluated, selected, and reused. Hence to explore the

role of community in ontology sharing and reuse,

participants were asked how interactions with people

in their domain may affect the way they tend to

evaluate an ontology for reuse. According to the

interviews, participants not only use the community

to evaluate an ontology before selecting it for reuse,

but some of them also evaluate the ontologies they

are building by the feedback they receive from the

community.

4.2.1 Build Related Information

Several researchers mentioned the importance of

different types of build related information such as

who/which organisation has built the ontology, what

the ontology has been built for e.g. the use case, who

are the different stakeholders of the ontology, how

the ontology was built (e.g. in collaboration), etc.

Interestingly, one of the first things interviewees

would say was that to evaluate an ontology, they

will ask themselves if they know the developer of

the ontology?

Interviewee BI3 for example emphasised the

importance of knowing the developer team and its

effect on the reuse process:

I have to say, in reusing thing, there is often

politics and connections are as important as

anything else. So, it is not always the best

one that wins.

He also added, quality of an ontology may

sometimes come second:

You know there might be constraint in

terms of I may not like a particular

ontology but because a bunch of other

people are using it and I want to standardize

with them, I might use it anyway.

Respondent SB4 also brought up the issue of

trusting the developer team:

Science is a social enterprise, I mean this is

how everything works in science, you know

if you look at a paper, do you trust the

paper? you look at the authors first and then

you read the paper and then you pick about

what they have done but yes I mean it is a

major criteria, major quality criteria, it may

or may not right; it is a bit of old boys club

but yes that is how people make decision. I

normally read the definitions and then go to

other things; do I trust the people who are

making it?

Besides the information about the developer

team or organisation, some of the respondents would

consider the reasons that ontology was built and

used for before selecting an ontology. They were

also interested in having some information about the

stakeholders of the ontologies. Interviewee SB3

said:

Completely separated from the people

developing it, are there other people who

uses this ontology beyond just that group,

that tells you something about it. I think

also finding out how they are using it, is

also important, you know what data is

being annotated with those ontology is also

important question, but I have some data

and I know I want to integrate with

something done in another institute, what is

the ontology there they are using, that is

also important, so I think there is a list of

the things you want to check!

4.2.2 Regular Updates and Maintenance

Ontology maintainability is one of the significant

metrics while evaluating the quality of an ontology

and before selecting it for reuse. In the interviews,

there were numerous examples of linking the quality

of an ontology to how regularly it is updated and

maintained. For some participants like NBI3,

regularity of updates was the first thing that they

would look at when evaluating a particular ontology:

Somebody build ontology during his

research in 1998 and he stored it on the web

and then he left it, it is available but not

updated, things will get obsolete very soon

so we make sure to use the ontologies

which are regularly updated, it is the first

thing.

Some of the respondents like SB3 compared

maintenance with some of the very popular quality

metrics in the literature like coverage and said:

Does it have my terms? I think is important

but there are many others that you need to

consider when you are picking an ontology

beyond just does it have the words in

ontology, about maintenance, do they

update regularly, do they release regularly?

do they have a record of doing that? How

responsive they are to updates when you

need new terms? all that sort of stuff. If

they are publishing it once every two years

it is probably not a good ontology.

Other participants like BI1 firmly believed that

updates and maintenance play a very important role

in their domain and said:

No way that an ontology is keeping on in

biology not getting updated, biology is

changing too fast so all the relevant

ontologies in biology are getting updated.

Interviewee NBI2 also made a link between the

nature of the domain that he is working in and the

necessity of regular updates:

It is about flexibility, if you want to, in

manufacturing business [towns] things are

changing all the time so you need solutions

that are easy and flexible to stay in, to stay

relevant to what you are doing tomorrow as

well as what you are trying to do today.

Interviewee BI3 compared the ontology

engineering with software engineering and said:

If you are going to reuse a piece of open

source software you will do the same thing,

you will open the GitHub website and say

you know if you looked in it and nobody

updated it or anything in three years, you

might think no; whereas if it looks like

there is an active ongoing community, you

will think yes, if I have problems I can ask

people and I can get bugs fixed.

BI4 believed that there is a link between the

popularity of an ontology and the regularity of

updating it and said:

It might be useful to use popular ones

because there are the ones that are mostly

updated so gene ontology has a release I

think every day or every 12 hours so the

popular ontologies are the ones that are

most updated.

Not only the regularity of updates is important, but

also how people deal with it is the other important

issue. Respondent SB3 talked about the importance

of having an update mechanism and said:

I think in the field that I am working, there

are other challenges, one of which is how

you deal with update mechanism of

ontologies, if you annotate data to ontology

which is typically use case for how you

keep up-to-date with the fact that ontologies

change reasonably often, you might have a

big database of data, that you used the data

in, new ontologies come along, the effect

the way the data has been represented in

your database, gotta have a update

mechanisms for dealing with that and that

can be tricky actually, it is not as simple

often as swapping things out when

something gets made obsolete, it is replaced

with other things, you have to deal with.

4.2.3 Responsiveness

Responsiveness of the ontology developer/

maintenance team was among one of the other

widely mentioned criteria when considering the

quality of ontologies for reuse. Some of the

respondents argued that not only knowing the

developer team or organization is important, but also

having an active ongoing community and their

willingness to collaborate, evolve and develop the

ontology further is an important factor when

assessing an ontology. Interviewee BI3 put it in this

way when he was asked about the importance of

responsiveness:

I would say it is definitely high up; I mean

having someone at the other end of line that

you feel that you can trust is definitely very

important. If it looks like there is an active

ongoing community, you will think yes if I

have problems I can ask people and I can

get bugs fixed.

Another respondent, SB5, used one of the

popular ontologies in her field as an example and

said:

For example, the fact that the Gene

ontology has a huge community behind it is

important because it means that they have a

curation process in place and quality

assurance and so on; so that kind of gives

more confidence that the ontology is as

good as it can be, it is not perfect for sure

but I mean that it is vetted by the

community.

Respondent BI1 chose responsiveness as the first

quality metric he would consider for evaluating an

ontology and compared it with one of the very

popular ontology evaluation metrics, that is

availability of documentation:

I would say the responsive of the team

obviously is the top-quality metric for me,

because nothing is perfect but if something

gets improved then it will get good like if

you have a question, you need to add a

term, something does not make sense, you

contact them, they answer and they answer

in a constructive way; this is good because

all the ontologies are work in progress,

there is no finished ontology in my domain.

4.2.4 Popularity

When asked about the popularity of an ontology and

its effect on quality evaluation, participants had

interesting thoughts and responses. As it was seen,

most of the interviewees defined popularity as the

number of times an ontology has been viewed or

used in a repository. The responses fall into three

different categories: those that were against the

metric, those who supported it and those who while

liking the popularity metric, did not agree with the

way it was being computed.

The first group of respondents thought the

popularity of an ontology considering the number of

times it has been used is not that important for them.

As interviewee BI1 would put it:

To me it would not be very important

except if two ontologies are really very

equal in everything else, I will take the

most used one but I do not think, it is not

really relevant to me, if it is the right tool

for the job, it is the right tool!

They also believed that the number of times an

ontology is used depends on its size, level of

specialization and the domain that it is built in and

cannot be considered as a metric to measure quality.

According to interviewee BI1:

Some ontologies are more specialized so

less people use them because it corresponds

to a very special need, but may this people,

are the right people and are using it well.

Interviewee SB3 also linked the use of an

ontology to its size and added:

If there is a small ontology but really

focused on representing an area that has not

been done before but it is correct, it is

absolutely correct, I think that is perfectly

reasonable, even if it is not widely used.

Some other interviewees like NBI5 found

popularity a helpful metric, but believed that it is

highly dependent on the domain that the ontology is

used in:

It depends on the domain that it has been

reused in, if it is just medical domain, it is

difficult to say that it is a reusable

ontology!

The second group agreed on the necessity of

having such a metric to identify the more popular

ontologies in different domains but were not sure

about the usefulness of the current methods that are

used to measure the popularity. As interviewee

NBI3 would put it:

How many times an ontology is viewed

will not help you, I may click just for

exploration, and I will say it is not my thing

and I don’t want it; it shows how catchy the

term is or how important, how regularly,

how often this term is chosen, but it does

not mean the use of the ontology; so, I

think there should be some other way.

BI4 used a very interesting personal experience

to prove the inaccuracy of the current techniques of

measuring the popularity:

When we were visualizing all the user

exploration on ontologies on BioPortal, and

we found that gene ontology is not accessed

that much using BioPortal and I thought

that it was very surprising because the gene

ontology is very famous and then I found

out because there is a gene ontology

browser called AmiGo, and their

visualizer tool is much better than

BioPortal visualisation of gene ontology, so

people generally go to gene ontology

website and lunch the AmiGo browser and

go to gene ontology there, so you can say

that gene ontology is much more accepted

but if you just look at the clicks (in

BioPortal) and you might say that gene

ontology is not that much famous.

Interviewee SB3 also thought that having a

quality metric like popularity is a step in the right

direction but believed that it might be misleading by

causing a snowball effect; according to him:

I can see that you can also putting a little

metric for usage or browsing or how many

people read these things, that is a kind of

useful but it does not tell you the whole

picture, you know you can end up with a

false signal there; you recommended an

ontology because it is useful because

someone uses it and then you recommend it

so someone else uses it and so on and so

on, what I mean, so you are getting in that

cycle of, it grows and grows!

The last and also the minority group were those

who thought it worth having a metric like popularity

and highlighted the importance of community

acceptance. According to interviewee NBI4:

If a community is using the ontology and is

happy with it I take thing to account so I try

to reuse or to do something to extend it or

maybe very careful on changing it. I need

to have motivations because after all

ontologies should have people working in

the domain and so if they are happy with

that one and I see things that are no good, I

point it out and I may suggest an extension,

whatever but I try to reuse what I have.

Some of the respondents brought up the other

definition that focuses on the link between

popularity and the number of imported ontologies.

NBI5 for example, made a link between the quality

of an ontology and the fact that the ontology has

reused other ontologies and said:

The quality of an ontology depends on the

relation between the ontology to upper

level ontologies; the more “same-as”,

“equivalent-as” links I can find in an

ontology. It also can be seen as a sign or a

feature of the ontology that can be reused

because if it is "same -as" a concept that we

already know, then it can be replaced.

NBI6 also believed that reusing some of the

ontologies are inevitable and not importing will

seem as a negative impression:

Whenever I have an ontology where there

is a person, I will never ever create my own

person class, I will always reuse FOAF. I

think it would be ridiculous to create my

own class and some of those are very very

strong class definition so it will always

worth reusing and I think it will be even

mistake by ontology engineer to develop

their own class and for me, if I see an

ontology doing that, I will get a negative

impression.

5 DISCUSSION

This paper set out to assess the process of evaluating

and selecting an ontology for reuse. Despite the

various evaluation methods and approaches

available in the literature, there is still no accepted

approach or a set of metrics that can be used to

evaluate ontologies effectively (Hlomani and Stacey,

2014) and most of the methods suffer from some

serious limitations (Lewen et al.,2006). When

respondents were asked how they evaluate an

ontology for reuse, they mentioned some of the very

well-known quality metrics based around content,

structure, and metadata of an ontology, but their

main focus was on the social and community aspects

around an ontology. The scope of this study has

particularly focused on the ways communities can

help the process of ontology selection and

evaluation.

As it was shown, most of the respondents

highlighted community related factors such as

reputation of the developer team or organisation in

the domain and regularity of updates as some very

important characteristics to be considered when

selecting an ontology for reuse. The study has also

found that the quality of ontologies is generally

considered to be limited and some have pointed out

that either way there isn’t such a thing as a complete

or finished ontology, hence ontologists often need to

count on the responsiveness of the ontology

developer team and organization as well as their

attitude toward the requests for changes. However,

this has not previously been described and most of

the existing studies have failed to cover and analyse

the interactions in the community that can help in

evaluating and selecting ontologies.

One unanticipated finding was that interviewees

suspected the usefulness of one of the most

commonly defined and used social quality metrics in

the literature, ‘popularity’. According to the

interviews, respondents care more about the projects

that the ontology has been or is being used in,

compared to the number of times it was used.

Regarding the second definition of popularity, that is

more about the linkage and the citation between

ontologies (Supekar et al.,2004), it seems that

further research should be undertaken to investigate

the importance of this factor and the way it can be

employed to calculate popularity of an ontology.

Overall, the evidence from this exploratory study

suggests that there is a clear interest for community

based ontology evaluation and the need for relevant

metrics. Further research is needed to confirm the

quality metrics suggested in these research

interviews and what their relative importance may

be, whether there are differences in ontology

engineering domains, or other important

idiosyncrasies deserving further attention. To

provide more generalizable findings for this

research, the next stage of our research agenda will

be to conduct large scale data collection via a survey

targeting ontology engineers from heterogeneous

domains. The expected outcome would be to

introduce a community based quality metrics as well

as to design and implement suggestions and

guidelines that will help in designing and

implementing ontologies that can be more easily

found and reused, based on community measures

identified through this ongoing research work.

6 CONCLUSIONS

This research study explored the set of steps

ontologists and knowledge engineers tend to take

when selecting an ontology for reuse. According to

the presented interview study, the process of

evaluating and selecting an ontology for reuse not

only depends on the ontology content and structure,

but it also depends on various non-ontological and

community related metrics, from how it was built to

how it has been maintained. Knowing about the

organisation and the developer team involved in

building and maintaining an ontology and their

responsiveness also seems to play an important role

in selecting and trusting an ontology. These findings

enhance extant understanding of the evaluation

metrics and it is hoped that they can be used to help

in the selection process. A natural progression of

this work is to design a framework based on non-

ontological and community based quality metrics for

ontology evaluation.

REFERENCES

Annamalai, M. and Sterling, L., 2003, July. Guidelines for

Constructing Reusable Domain Ontologies. In OAS

(pp. 71-74).

Arpinar, I.B., Giriloganathan, K. and Aleman-Meza, B.,

2006, May. Ontology quality by detection of conflicts

in metadata. In Proceedings of the 4th International

EON Workshop.

Bontas, E.P., Mochol, M. and Tolksdorf, R., 2005, June.

Case studies on ontology reuse. In Proceedings of the

IKNOW05 International Conference on Knowledge

Management (Vol. 74).

Brank, J., Grobelnik, M. and Mladenić, D., 2005. A survey

of ontology evaluation techniques.

Bürger, T. and Simperl, E., 2008. Measuring the benefits

of ontologies. In On the Move to Meaningful Internet

Systems: OTM 2008 Workshops (pp. 584-594).

Springer Berlin/Heidelberg.

Burton-Jones, A., Storey, V.C., Sugumaran, V. and

Ahluwalia, P., 2005. A semiotic metrics suite for

assessing the quality of ontologies. Data & Knowledge

Engineering, 55(1), pp.84-102.

Ding, Y. and Foo, S., 2002. Ontology research and

development. part 2-a review of ontology mapping and

evolving. Journal of information science, 28(5),

pp.375-388.

Fernández, M., Overbeeke, C., Sabou, M. and Motta, E.,

2009, December. What makes a good ontology? A

case-study in fine-grained knowledge reuse. In Asian

Semantic Web Conference (pp. 61-75). Springer Berlin

Heidelberg.

Guest, G., Bunce, A. and Johnson, L., 2006. How many

interviews are enough? An experiment with data

saturation and variability. Field methods, 18(1), pp.59-

82.

Hlomani, H. and Stacey, D., 2014. Approaches, methods,

metrics, measures, and subjectivity in ontology

evaluation: A survey. Semantic Web Journal, pp.1-5.

Lewen, H., Supekar, K., Noy, N.F. and Musen, M.A.,

2006, May. Topic-specific trust and open rating

systems: An approach for ontology evaluation. In

Workshop on Evaluation of Ontologies for the Web.

Lozano-Tello, A. and Gómez-Pérez, A., 2004. Ontometric:

A method to choose the appropriate ontology. Journal

of database management, 2(15), pp.1-18.

Martínez-Romero, M., Jonquet, C., O’Connor, M.J.,

Graybeal, J., Pazos, A. and Musen, M.A., 2017.

NCBO Ontology Recommender 2.0: an enhanced

approach for biomedical ontology recommendation.

Journal of biomedical semantics, 8(1), p.21.

McDaniel, M., Storey, V.C. and Sugumaran, V., 2016,

June. The Role of Community Acceptance in

Assessing Ontology Quality. In International

Conference on Applications of Natural Language to

Information Systems (pp. 24-36). Springer

International Publishing.

Page, L., Brin, S., Motwani, R. and Winograd, T., 1999.

The PageRank citation ranking: Bringing order to the

web. Stanford InfoLab.

Shadbolt, N., Berners-Lee, T. and Hall, W., 2006. The

semantic web revisited.IEEE intelligent systems,

21(3), pp.96-101.

Simperl, E., 2009. Reusing ontologies on the Semantic

Web: A feasibility study. Data & Knowledge

Engineering, vol. 68, no. 10, pp.905-925.

Supekar, K., Patel, C. and Lee, Y., 2004, May.

Characterizing Quality of Knowledge on Semantic

Web. In FLAIRS Conference (pp. 472-478).

Tartir, S., Arpinar, I.B. and Sheth, A.P., 2010. Ontological

evaluation and validation. In Theory and applications

of ontology: Computer applications (pp. 115-130).

Springer Netherlands.

Tello, A.J.L., 2002. Métrica de idoneidad de ontologías

(Doctoral dissertation, Universidad de Extremadura).

Uschold, M., Healy, M., Williamson, K., Clark, P. and

Woods, S., 1998, June. Ontology reuse and

application. In Formal ontology in information

systems, vol. 179, p. 192.

Wang, X., Guo, L. and Fang, J., 2008, April. Automated

ontology selection based on description logic. In

Computer Supported Cooperative Work in Design,

2008. CSCWD 2008. 12th International Conference

on (pp. 482-487). IEEE.

Yu, J., Thom, J.A. and Tam, A., 2009. Requirements-

oriented methodology for evaluating ontologies.

Information Systems, 34(8), pp.766-791.