Semantic Similarity between Queries in QA System using a

Domain-speciﬁc Taxonomy

Hilda Kosorus

, Andreas B

ogl

and Josef K

ung

Institute of Applied Knowledge Processing, Johannes Kepler University, Altenbergerstraße 69, Linz, Austria

MEOVI, Hagenberg, Austria

Keywords:

Query recommendation, Semantic similarity, Short text similarity, Taxonomy.

Abstract:

Semantic similarity has been extensively studied in the past decades and has become a rapidly growing ﬁeld of

research. Sentence or short text similarity measures play an important role in text-based applications, such as

text mining, information retrieval and question answering systems. In this paper we consider the problem of se-

mantic similarity between queries in a question answering system with the purpose of query recommendation.

Our approach is based on an existing domain-speciﬁc taxonomy. We deﬁne innovative three-layered semantic

similarity measures between queries using existing similarity measures between ontology concepts combined

with various set-based distance measures. We then analyse and evaluate our approach against human intu-

ition using a data set of 90 questions. Further on, we argue that these measures are taxonomy-dependent and

are inﬂuenced by various factors: taxonomy structure, keyword mappings, keyword weights, query-keyword

mappings and the chosen concept similarity measure.

1 INTRODUCTION

Current implementations of QA systems that incor-

porate a recommendation mechanism are based on (i)

methods using external sources, like user proﬁles, (ii)

methods based on expectations (e.g. query patterns,

models) or (iii) methods using query logs (Marcel and

Negre, 2011). These methods do not take into ac-

count the semantic meaning of queries. In the past

two decades researchers have been studying seman-

tic similarity in order to improve information retrieval

and develop intelligent semantic systems.

A semantic sentence similarity measure can have

an important role in the development of a query re-

commender system. Nevertheless, such measures can

be successfully used in other directions, like query

clustering for discovering “hot topics” or to ﬁnd the

query that best represents a cluster, pattern recogni-

tion for identifying user groups or in web page re-

trieval to calculate page title similarities.

Studies of semantic similarity in the past decades

has been focusing on two extremes: either measu-

ring the similarity between single words or concepts

or between documents. However, there is a growing

need for an effective method to compute short text si-

milarity. Web search technologies incorporate tasks,

such as query reformulation, query recommendation,

sponsored search and image retrieval, that rely on ac-

curately computing similarity between two very short

segments of text. Unfortunately, traditional tech-

niques for detecting similarity between documents

and queries fail when directly applied to these tasks.

Such methods rely on analysing shared words or the

co-occurence of terms in both the query and the doc-

ument.

In this paper we deﬁne innovative three-layered

semantic similarity measures between queries using

existing similarity measures between ontology con-

cepts combined with various set-based distance mea-

sures. We then analyze and evaluate our approach

against human intuition using a dataset of 90 ques-

tions. The goal of this paper is to present semantic

query similarity measures that can be successfully in-

tegrated into query recommender systems and to eval-

uate and compare them in terms of human judgement.

The rest of the paper is structured as follows. In

section 2 we review related work in the area of seman-

tic similarity measures between concepts, between

sets of concepts and the area of short text similarity.

In section 3 we present and deﬁne the domain-speciﬁc

taxonomy on which our semantic similarity measures

are based. In section 4 we introduce similarity mea-

sures between queries as a combination of topic si-

milarity and keyword similarity using the deﬁned ta-

241

Kosorus H., Bögl A. and Küng J..

Semantic Similarity between Queries in QA System using a Domain-speciﬁc Taxonomy.

DOI: 10.5220/0003965902410246

In Proceedings of the 14th International Conference on Enterprise Information Systems (ICEIS-2012), pages 241-246

ISBN: 978-989-8565-10-5

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

xonomy. In section 5 we analyze and evaluate these

similarity measures. Finally, in section 6 we summa-

rize the contents of this paper, drawing some impor-

tant conclusions and present our future work.

2 RELATED WORK

The problem of similarity is a heavily researched sub-

ject in particular in information retrieval, but also in

general in computer science, artiﬁcial intelligence,

philosophy and natural language processing. Mea-

suring similarity between documents has a long tra-

dition in information retrieval, but these approaches

compare only vectors of document features (Burgess

et al., 1998; Landauer et al., 1998a; Landauer et al.,

1998b), usually single words or word stems, by coun-

ting their occurrence in the document.

There is extensive literature on measuring simi-

larity between concepts within a taxonomy (Rada

et al., 1989; Lee et al., 1993; Wu and Palmer, 1994;

Resnik, 1995; Jiang and Conrath, 1997; Leacock and

Chodorow, 1998; Lin, 1998; Resnik, 1999; Li et al.,

2003; Bouquet et al., 2004; Haase et al., 2004; Cord

et al., 2005; Al-Mubaid and Nguyen, 2006; Wang

et al., 2006; Lee et al., 2008; Dong et al., 2009; Bin

et al., 2009), while there are few publications that

cover the area of short text semantic similarity (Li

et al., 2006; O’Shea et al., 2010; Oliva et al., 2011)

and some related to semantic similarity between sets

of concepts (Bouquet et al., 2004; Haase et al., 2004;

Cord

ı et al., 2005). In (Li et al., 2006) it is argued

that existing long text similarity measures have some

limitations and drawbacks and their performance is

unsatisfactory when applied to short sentences.

In the following we will brieﬂy present the related

research in the domain of semantic similarity between

concepts and between sets of concepts.

2.1 Semantic Similarity between

Concepts using Taxonomies

There are basically two ways of using an ontology or

taxonomy to determine the semantic similarity bet-

ween concepts: the edge-based approach and the

information content-based approach (Resnik, 1995;

Resnik, 1999; Lin, 1998). In the following we will

make a short overview of the edge-based approaches.

Intuitively, the similarity of different concepts in

an ontology is measured by computing the distance

within the ontology. Namely, if two concepts reside

closer in the ontology, then we can conclude that they

are more similar. When computing the ontology dis-

tance we actually use the specialization graph of ob-

jects and we deﬁne it as being the shortest path bet-

ween the two concepts (Rada et al., 1989).

Rada, Mili, Bicknell and Blettner (1989) deﬁned

the conceptual distance as

sim(c

, c

) = minimum number o f edges

separating c

and c

where c

and c

are the node representation of the two

concepts in the ontology. Wu and Palmer (2004) re-

deﬁned the edge-based similarity measure taking into

account the depth of the nodes in the hierarchical

graph:

sim(c

, c

) =

2 ×N

+ N

+ 2 × N

, (1)

where N

and N

are the number of nodes from c

and

, respectively, to c

, the least common superconcept

(LCS) of c

and c

, and N

is the number of nodes on

the path from c

to the root node.

Li et al. (2003) deﬁned the similarity between two

concepts as:

sim(c

, c

) =

(

−αl

βh

−e

−βh

βh

−βh

, if c

6= c

1 , otherwise

(2)

where, similarly, the parameters α and β scale the

contribution of the two values l = N

and h = N

Based on the benchmark data set, they obtained the

optimal parameters α = 0.2 and β = 0.6.

2.2 Semantic Similarity between Sets of

Concepts

Deﬁning a semantic similarity measure between sets

of concepts was the next step in computing semantic

similarity mainly for information retrieval purposes.

In (Bouquet et al., 2004) the ontological distance

between sets of concepts is computed by summing

up the distances between every pair (c

, c

), where

∈ C

and c

∈ C

. Haase et al. (2004) used the

edge-based similarity measure between concepts de-

ﬁned by Li et al. (2006) (see 2) to introduce the simi-

larity between sets of concepts as:

Sim(C

, C

) =

∑

∈C

max

∈C

sim(c

, c

), (3)

which computes an average of distances between c

∈

and the most similar concept in C

In (Cord

ı et al., 2005) a new similarity measure

between sets of concepts was introduced, which gives

more weight to keyword pairs with a higher similarity,

but still allowing lower values to contribute to the ﬁnal

outcome.

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

242

Figure 1: Snapshot of the topic-tree with keywords and their

weights.

3 THE DOMAIN-SPECIFIC

TAXONOMY

Before introducing our proposed semantic query si-

milarities, it is important to understand the structure

of the underlying domain-speciﬁc taxonomy. While

most of the previously described similarity measures

make use of the english lexical taxonomy WordNet

our similarity measures are based on a new domain-

speciﬁc (nutrition) taxonomy with a tree-like struc-

ture, where the links between nodes represent IS-A

relationships. In the following we will refer to this

structure as ”topic-tree”.

Our topic-tree is composed of a set of topics:

T = {t

, t

, ..., t

an IS-A relationship between topics:

L ⊂ T × T , (t

, t

) ∈ L ⇐⇒ t

parent o f t

a set of keywords:

K = {k

, k

, ..., k

a mapping relationship between topics and keywords:

M ⊂ T ×K , (t

, k

) ∈ M ⇐⇒ k

mapped to t

and the corresponding mapping weights:

w : M → (0, 1],

where the value w(t

, k

) represents how relevant is

keyword k

for topic t

Figure 1 shows a partial snapshot of the above de-

ﬁned taxonomy. The topics represent selected cate-

gories and sub-categories in the speciﬁed domain (i.e.

nutrition), the mapped keywords are frequent rele-

vant words occuring within these topics which were

obtained by crawling related websites and/or docu-

ments. The corresponding weights were calculated

using the TF-IDF method (Salton and Buckley, 1988).

http://wordnet.princeton.edu/

4 PROPOSED SEMANTIC

SIMILARITY MEASURES

Let Q = {q

, q

, ..., q

} be a set of queries in the nu-

trition domain. We want to deﬁne a semantic simi-

larity measure sim

: Q × Q → [0, 1] between these

queries using the topic-tree deﬁned in section 3. We

assume that to each query q ∈ Q we can assign a set

of keywords S

⊂ K , where S

was extracted from

q using some natural language processing methods

(HaCohen-Kerner et al., 2005; Turney, 2000; Hulth,

2003). For example, for

q = “What type o f f ood can I eat and at what time

in order to lose weight?

= { f ood, eat, time, lose weight}.

In the following we will deﬁne the semantic query

similarity sim

using three other similarity measures:

between topics, between keywords and between sets

of keywords, each incorporating the one before.

4.1 Semantic Similarity between Topics

Let sim

: T × T → [0, 1] be the topic similarity func-

tion where sim

, t

) represents the semantic simi-

larity between two topics t

, t

∈ T using the struc-

ture of the topic-tree. For our experiments, we deﬁned

sim

using the similarity measures (1) and (2).

4.2 Semantic Similarity between

Keywords

Let sim

: K × K → [0, 1] be the keyword similarity

function where sim

, k

) represents the semantic

similarity between two keywords k

, k

∈ K . We de-

ﬁne sim

in the following way:

sim

, k

) =

+ w

sim

, t

) (4)

where

= max

(t,k

)∈M

w(t, k

), i ∈ {p, q}

and

= arg max

(t,k

)∈M

w(t, k

), i ∈ {p, q}.

4.3 Semantic Similarity between Sets of

Keywords

Let sim

: P (K ) × P (K ) → [0, 1] be the keyword-

set similarity function where sim

, S

) represents

the semantic similarity between two sets of keywords

, S

⊂ K and P (K ) contains all subsets of K . In

the following we will introduce several possible de-

ﬁnitions of sim

using well-known set distance mea-

sures from the literature.

SemanticSimilaritybetweenQueriesinQASystemusingaDomain-specificTaxonomy

243

4.3.1 The Sum of Maximum Similarities

The sum of minimum distances measure was origi-

nally deﬁned by Niiniluoto (1987) to measure truth-

likeness in belief revision theory. We apply the same

concept to deﬁne the similarity measure sim

bet-

ween sets of keywords (the sum of maximum simila-

rities):

sim

, S

) =



∑

∈S

Sim(k

, S

) +

∑

∈S

Sim(k

, S

)



(5)

where

Sim : K ×P (K ) → [0, 1], Sim(k, S) = max

∈S

sim

(k, k

is the semantic similarity between a keyword k ∈ K

and a set of keywords S ⊂ K .

4.3.2 The Surjection Measure

The surjection measure was introduced by Oddie

(1979), who suggested deﬁning the distance between

two sets by considering surjections that map the larger

set to the smaller one. We applied this concept to mea-

sure similarity between sets of keywords, and deﬁned

surjection similarity measure, sim

, as

sim

, S

) = max

|η|

∑

)∈η

sim

, k

). (6)

where the maximum is taken over all surjections η

that maps the larger set to the smaller one.

4.3.3 The Maximum Link Similarity Measure

The minimum link distance measure was proposed in

(Eiter and Mannila, 1997) as an alternative to the pre-

viously mentioned distance measures between point

sets. First, let us deﬁne the linking between S

and S

as a relation R ⊆ S

× S

satisfying

(a) for all k

∈ S

there exists k

∈ S

such that

, k

) ∈ R

and

(b) for all k

∈ S

there exists k

∈ S

such that

, k

) ∈ R .

We now apply this concept to deﬁne the maximum link

similarity between sets of keywords as

sim

, S

) = max

|R |

∑

)∈R

sim

, k

), (7)

taking the maximum over all relations R .

4.4 Semantic Similarity between

Queries

Finally, we deﬁne the query similarity measure sim

Q ×Q → [0, 1] as

sim

, q

) = sim

, S

) (8)

where S

, S

⊂ K are the corresponding set of key-

words extracted from q

and q

, respectively.

5 COMPARISON AND

EVALUATION

In order to evaluate these similarity measures we con-

ducted a survey with 15 persons, men and women,

age between 25 and 60. We randomly sampled 50

pairs from a dataset of 90 different questions in the

nutrition domain and asked the survey participants to

compare and measure the relatedness of each pair by

ranking them with a value between 0 and 4 (0=not

related at all, 1=somehow related, 2=related, 3=very

related, 4=similar).

Finally, we compared the participants’ ranking

against six different semantic similarity measures: the

one deﬁned by Haase et al. (3), the sum of all simi-

larities (Bouquet et al., 2004), the one introduced by

Cord

ı (2005), the cosine similarity (Li et al., 2003),

the sum of maximum similarities (5), the surjection

similarity (6) and the maximum link similarity (7).

While some question pairs were ranked almost the

same by all participants (low variance), there were

some cases where participants answered very diffe-

rently (high variance). This reﬂects how diversely is

the “relatedness” of two questions perceived by hu-

mans. Table 1 contains the mean, maximum and mini-

mum variances calculated by question pairs rankings.

Table 2 contains the correlation values of each se-

mantic similarity method with the average participant

ranking values.

Table 1: Survey results - Variances calculated by question

pair rankings.

Mean variance 0.93

Maximum variance 2.14

Minimum variance 0

Based on our experiments and the above results

we make the following observations:

• The semantic similarity measures depend on the

structure of the taxonomy (Bernstein et al., 2005).

In our case, the topic hierarchy, the keyword-topic

mappings and the assigned keyword weights af-

fect the computed similarity.

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

244

Table 2: Correlation between survey results and the seman-

tic similarity measures.

Method Correlation

Haase 0.605

Sum of All 0.597

Cord

ı 0.563

Cosine 0.563

Sum of Maximum 0.617

Surjection 0.634

Maximum Link 0.626

• The similarity measure between sets of keywords,

and therefore between queries, depends on the

chosen topic similarity (edge-based or informa-

tion content-based) and on the keyword similarity.

In our experiments we used the edge-based simi-

larity measures deﬁned by Wu and Palmer (1994)

and Li et al. (2003).

Table 3: Types of question pairs based on ranking variance

and difference between average survey ranking and seman-

tic similarity values.

Type Var. Diff. Percentage

A low low 48%

B high low 20%

C low high 12%

D high high 20%

• Although the correlation between the participants’

ranking and the evaluated measures are rather low

(see table 2), this can be explained by the follow-

ing factors:

– the queries are selected from a speciﬁc and nar-

row domain (nutrition),

– the concepts that appear in the queries are

rather complex,

– the participants’ ranking for some question

pairs was very diverse,

– the participants tend to understand the ranking

values or the question pair “relatedness” diffe-

rently.

• The correlation results (between 0.563 and 0.634)

do not contradict the fact that the semantic simi-

larity measures reﬂect on some level the human

perception. Most of the question pairs were eval-

uated by the participants and the semantic simila-

rity measures almost the same. In our evaluation,

compared to the surjection measure, 48% of the

question pairs were of type A and 20% of type B

(see table 3).

6 CONCLUSIONS AND FUTURE

WORK

In this paper we introduced innovative three-layered

semantic similarity measures between queries using

a domain-speciﬁc taxonomy. We evaluated our mea-

sures by conducting an on-line survey and compar-

ing them and other four existing semantic similarity

measures against the participants’ intuition. The re-

sults show that our similarity measures have a higher

correlation with the average survey ranking than the

other four measures. We believe that measuring se-

mantic similarity between concepts using taxonomies

can improve signiﬁcantly the results retrieved by re-

commender systems. We also argue that these mea-

sures depend on the structure of the underlying taxo-

nomy (hierarchy, keyword-topic mappings, keyword

weights, etc.) and on the chosen concept-to-concept

similarity measure. In the future, we plan to analyze

the aspects that alter the behavior of the semantic si-

milarity measures.

In this context, we distinguish two types of recom-

mendations. The ﬁrst type can be directly obtained by

using the semantic similarity measure and retrieving

the queries with the highest similarity to the user’s last

query. These recommendations will be rather “gene-

ral” and maybe “too similar” to the last query (i.e.

predictions with low probability). The second type of

recommendations requires a much elaborate analysis

(extracting patterns, clustering) of all users’ history

and then comparing the learned query patterns to the

current user’s history. With this type of recommenda-

tions we can predict the user’s next set of questions

(with a high probability) and, on the long run, his in-

terests and goals. In the future we intend to focus on

the second type of recommendations. We also plan to

test the goodness of the semantic recommendations

by analyzing users’ feedback.

ACKNOWLEDGEMENTS

The authors would like to thank MEOVI

for the ﬁ-

nancial support during their research that lead to the

ﬁndings presented in this paper.

REFERENCES

Al-Mubaid, H. and Nguyen, H. A. (2006). A cluster-

based approach for semantic similarity in the biome-

dical domain. In Proceedings of the 28th IEEE EMBS

www.meovi.com

SemanticSimilaritybetweenQueriesinQASystemusingaDomain-specificTaxonomy

245

Annual International Conference, pages 2713–2717,

New York, USA.

Bernstein, A., Kaufmann, E., B

urki, C., and Klein, M.

(2005). How similar is it? Towards personalized si-

milarity measures in ontologies. In 7. Internationale

Tagung Wirtschaftsinformatik, pages 1347–1366.

Bin, S., Liying, F., Jianzhuo, Y., Pu, W., and Zhongcheng,

Z. (2009). Ontology-based measure of semantic simi-

larity between concepts. In World Congress on Soft-

ware Engineering, volume 2, pages 109–112.

Bouquet, P., Kuper, G., Scoz, M., and Zanobini, S. (2004).

Asking and answering semantic queries. In Proceed-

ings of Meaning Coordination and Negotiation Work-

shop (MCNW-04) in conjunction with International

Semantic Web Conference.

Burgess, C., Livesay, K., and Lund, K. (1998). Explorations

in context space: Words, sentences, discourse. Dis-

course Processes, 25(2-3):211–257.

Cord

ı, V., Lombardi, P., Martelli, M., and Mascardi, V.

(2005). An ontology-based similarity between sets of

concepts. In 6th Joint Workshop ”From Objects to

Agents”: Simulation and Formal Analysis of Complex

Systems, pages 16–21, Camerino,Italy.

Dong, H., Hussain, F. H., and Chang, E. (2009). A hybrid

concept similarity measure model for ontology envi-

ronment. In Proceedings of the Confederated Interna-

tional Workshops and Posters on the Move to Mean-

ingful Internet Systems, pages 848–857.

Eiter, T. and Mannila, H. (1997). Distance measures for

point sets and their computation. Journal Acta Infor-

matica, 34:103–133.

Haase, P., Siebes, R., and Harmelen, F. V. (2004). Peer se-

lection in peer-to-peer networks with semantic topolo-

gies. In International Conference on Semantics of a

Networked World: Semantics for Grid Databases.

HaCohen-Kerner, Y., , Gross, Z., and Masa, A. (2005). Au-

tomatic extraction and learning of keyphrases from

scientiﬁc articles. In Gelbukh, A., editor, Compu-

tational Linguistics and Intelligent Text Processing,

volume 3406 of Lecture Notes in Computer Science,

pages 657–669. Springer Berlin / Heidelberg.

Hulth, A. (2003). Improved automatic keyword extraction

given more linguistic knowledge. In Proceedings of

the 2003 conference on Empirical methods in nat-

ural language processing, EMNLP ’03, pages 216–

223, Stroudsburg, PA, USA. Association for Compu-

tational Linguistics.

Jiang, J. and Conrath, W. (1997). Semantic similarity based

on corpus statistics and lexical taxonomy. In Proceed-

ings of International Conference Research on Compu-

tational Linguistics, pages 19–33, Taiwan.

Landauer, T. K., Foltz, P. W., and Laham, D. (1998a). In-

troduction to latent semantic analysis. Discourse Pro-

cesses, 25(2-3):259–284.

Landauer, T. K., Laham, D., and Foltz, P. (1998b). Learning

human-like knowledge by singular value decomposi-

tion: A progress report. In Advances in Neural In-

formation Processing Systems 10, pages 45–51. MIT

Press.

Leacock, C. and Chodorow, M. (1998). Combining Local

Context and WordNet Similarity for Word Sense Iden-

tiﬁcation, pages 305–332. In C. Fellbaum (Ed.), MIT

Press.

Lee, J. H., Kim, M. H., and Lee”, Y. J. (1993). Information

retrieval based on conceptual distance in IS-A hierar-

chies. Journal of Documentation, 49(2):188–207.

Lee, W. N., Shah, N., Sundlass, K., and Musen, M.

(2008). Comparison of ontology-based semantic-

similarity measures. In AMIA Annual Symposium Pro-

ceedings, pages 384–388.

Li, Y., Bandar, Z. A., and McLean, D. (2003). An approach

for measuring semantic similarity between words us-

ing multiple information sources. IEEE Transactions

on Knowledge and Data Engineering, 15(4).

Li, Y., McLean, D., Bandar, Z. A., O’Shea, J. D., and

Crockett, K. (2006). Sentence similarity based on se-

mantic nets and corpus statistics. IEEE Transactions

on Knowledge and Data Engineering, 18(8):1138–

1150.

Lin, D. (1998). An information-theoretic deﬁnition of simi-

larity. In Proceedings of the 15th International Con-

ference on Machine Learning, pages 296–304.

Marcel, P. and Negre, E. (2011). A survey of query recom-

mendation techniques for data warehouse exploration.

emes Journ

ees Francophones sur les Entrep

ots de

Donn

ees et l’Analyse en ligne (EDA), B-7.

Oliva, J., Serrano, J. I., del Castillo, M. D., and Iglesias, A.

(2011). Sysmss: A syntax-based measure for short-

text semantic similarity. Data and Knowledge Engi-

neering, 70:390–405.

O’Shea, J., Bandar, Z., Crockett, K., and McLean, D.

(2010). Benchmarking short text semantic similarity.

International Journal of Intelligent Information and

Database Systems, 4(2):103 – 120.

Rada, R., Mili, H., Bicknell, E., and Blettner, M. (1989).

Development and application of a metric on semantic

nets. IEEE Transactions on Systems, Man, and Cyber-

netics, 19(1):17–30.

Resnik, P. (1995). Using information content to evaluate

semantic similarity in a taxonomy. In Proceedings of

IJCAI-95, pages 448–453, Montreal, Canada.

Resnik, P. (1999). Semantic similarity in a taxonomy:

An information-based measure and its application to

problems of ambiguity in natural language. Journal

or Artiﬁcial Intelligence Research, 11:95–130.

Salton, G. and Buckley, C. (1988). Term-weighting ap-

proaches in automatic text retrieval. In Information

Processing and Management, pages 513–523.

Turney, P. D. (2000). Learning algorithms for keyphrase

extraction. Information Retrieval, 2(4):303–336.

Wang, G. H., Wang, Y. D., and Guo, M. Z. (2006). An

ontology-based method for similarity calculation of

concepts in the semantic web. In Proceedings of the

5th International Conference on Machine Learning

and Cybernetics, pages 1538–1542, Dalian, China.

Wu, Z. and Palmer, M. (1994). Verb semantics and lexical

selection. In 32nd Annual Meeting of the Association

for Computational Linguistics, pages 133–138.

ICEIS2012-14thInternationalConferenceonEnterpriseInformationSystems

246