Jaimie Murdock, Cameron Buckner, Colin Allen


Ontology evaluation poses a number of difficult challenges requiring different evaluation methodologies, particularly for a “dynamic ontology” representing a complex set of concepts and generated by a combination of automatic and semi-automatic methods. We review evaluation methods that focus solely on syntactic (formal) correctness, on the preservation of semantic structure, or on pragmatic utility. We propose two novel methods for dynamic ontology evaluation and describe the use of these methods for evaluating the different taxonomic representations that are generated at different times or with different amounts of expert feedback. The proposed “volatility” and “violation” scores represent an attempt to merge syntactic and semantic considerations. Volatility calculates the stability of the methods for ontology generation and extension. Violation measures the degree of “ontological fit” to a text corpus representative of the domain. Combined, they support estimation of convergence towards a stable representation of the domain. No method of evaluation can avoid making substantive normative assumptions about what constitutes “correct” representation, but rendering those assumptions explicit can help with the decision about which methods are appropriate for selecting amongst a set of available ontologies or for tuning the design of methods used to generate a hierarchically organized representation of a domain.


  1. Brank, J., Grobelnik, M., and Mladenic, D. (2005). Survey of ontology evaluation techniques. In Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD).
  2. Brewster, C., Alani, H., Dasmahapatra, S., and Wilks, Y. (2004). Data driven ontology evaluation. In Proceedings of LREC, volume 2004.
  3. Buckner, C., Niepert, M., and Allen, C. (2010). From encyclopedia to ontology: Toward dynamic representation of the discipline of philosophy. Synthese.
  4. Dellschaft, K. and Staab, S. (2008). Strategies for the Evaluation of Ontology Learning. In Buitelaar, P. and Cimiano, P., editors, Ontology Learning and Population: Bridging the Gap Between Text and Knowledge, pages 253-272. IOS Press.
  5. Eckert, K., Niepert, M., Niemann, C., Buckner, C., Allen, C., and Stuckenschmidt, H. (2010). Crowdsourcing the Assembly of Concept Hierarchies. In Proceedings of the 10th ACM/IEEE Joint Conference on Digital Libraries (JCDL), Brisbane, Australia. ACM Press.
  6. Fahad, M. and Qadir, M. (2008). A Framework for Ontology Evaluation. In Proceedings International Conference on Conceptual Structures (ICCS), Toulouse, France, July, pages 7-11. Citeseer.
  7. Gangemi, A., Catenacci, C., Ciaramita, M., and Lehmann, J. (2006). Modelling ontology evaluation and validation. In The Semantic Web: Research and Applications, pages 140-154. Springer.
  8. Gómez-Pérez, A. (1999). Evaluation of taxonomic knowledge in ontologies and knowledge bases. In Proceedings of the 12th Banff Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Alberta, Canada.
  9. Gruber, T. R. (1995). Toward principles for the design of ontologies used for knowledge sharing. International Journal of Human Computer Studies, 43(5):907-928.
  10. Guarino, N. and Welty, C. A. (2004). An overview of OntoClean. In Staab, S. and Studer, R., editors, Handbook on ontologies, chapter 8, pages 151-159. Springer, 2 edition.
  11. Jiang, J. and Conrath, D. (1997). Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of International Conference Research on Computational Linguistics (ROCLING X), number Rocling X, Taiwan.
  12. Kuhn, T. (1962). The Structure of Scientific Revolutions. University of Chicago Press.
  13. Lin, D. (1998). An information-theoretic definition of similarity. In Proceedings of the 15th International Conference on Machine Learning, pages 296-304. Citeseer.
  14. Lozano-Tello, A. and Gómez-Pérez, A. (2004). Ontometric: A method to choose the appropriate ontology. Journal of Database Management, 15(2):1-18.
  15. Maedche, A. and Staab, S. (2002). Measuring similarity between ontologies. Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web, pages 15-21.
  16. Niepert, M., Buckner, C., and Allen, C. (2007). A dynamic ontology for a dynamic reference work. In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries, page 297. ACM.
  17. Niepert, M., Buckner, C., and Allen, C. (2008). Answer set programming on expert feedback to populate and extend dynamic ontologies. In Proceedings of 21st FLAIRS.
  18. Noy, N. and McGuinness, D. (2001). Ontology development 101: A guide to creating your first ontology.
  19. Porzel, R. and Malaka, R. (2005). A task-based framework for ontology learning, population and evaluation. In Buitelaar, P., Cimiano, P., and Magnini, B., editors, Ontology Learning from Text: Methods, Evaluation and Applications. IOS Press, Amsterdam.
  20. Resnik, P. (1999). Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. Journal of artificial intelligence research, 11(4):95-130.
  21. Shannon, C. E. (1949). A mathematical theory of communication. University of Illinois Press, Urbana, Illinois.
  22. Smith, B. (2003). Ontology. In Luciano, F., editor, Blackweel Guide to the philosophy of computing and information, pages 155-166. Blackwell, Oxford.
  23. Smyth, P. and Goodman, R. (1992). An information theoretic approach to rule induction from databases. IEEE Transactions on Knowledge and Data Engineering, 4(4):301-316.
  24. Staab, S., Gómez-Pérez, A., Daelemans, W., Reinberger, M.-L., Guarino, N., and Noy, N. F. (2004). Why evaluate ontology technologies? because it works! IEEE Intelligent Systems, 19(4):74-81.
  25. Supekar, K. (2004). A peer-review approach for ontology evaluation. In 8th Int. Protege Conf, pages 77-79. Citeseer.
  26. Velardi, P., Navigli, R., Cucchiarelli, A., and Neri, F. (2005). Evaluation of OntoLearn, a methodology for automatic learning of domain ontologies. In Buitelaar, P., Cimiano, P., and Magnini, B., editors, Ontology Learning from Text: Methods, Evaluation and Applications. IOS Press, Amsterdam.

Paper Citation

in Harvard Style

Murdock J., Buckner C. and Allen C. (2010). TWO METHODS FOR EVALUATING DYNAMIC ONTOLOGIES . In Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2010) ISBN 978-989-8425-29-4, pages 110-122. DOI: 10.5220/0003101601100122

in Bibtex Style

author={Jaimie Murdock and Cameron Buckner and Colin Allen},
booktitle={Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2010)},

in EndNote Style

JO - Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2010)
SN - 978-989-8425-29-4
AU - Murdock J.
AU - Buckner C.
AU - Allen C.
PY - 2010
SP - 110
EP - 122
DO - 10.5220/0003101601100122