Knowledge Resource Development for Identifying Matching Image Descriptions

Alicia Sagae, Scott E. Fahlman


Background knowledge resources contribute to the performance of many current systems for textual inference tasks (QA, textual entailment, summarization, retrieval, and others). However, it can be difficult to assess how additions to such a knowledge base will impact a system that relies on it. This paper describes the incremental, task-driven development of an ontology that provides features to a system that retrieves images based on their textual descriptions. We perform error analysis on a baseline system that uses lexical features only, then focus ontology development on reducing these errors against a development set. The resulting ontology contributes more to performance than domain-general resources like WordNet, even on a test set of previously unseen examples.


  1. Blei, D. and Jordan, M. (2003). Modeling annotated data. In the 26th annual International ACM SIGIR Conference on Research and Development in Information Retrieval, page 127134. ACM Press.
  2. Collins, M. (2002). Discriminative training methods for hidden markov models: theory and experiments with perceptron algorithms. In EMNLP 7802: Proceedings of the ACL-02 conference on Empirical methods in natural language processing, pages 1-8, Morristown, NJ, USA. Association for Computational Linguistics.
  3. Csurka, G., Clinchant, S., and Popescu, A. (2011). XRCE and CEA LIST's Participation at Wikipedia Retrieval of ImageCLEF 2011. In Petras, V., Forner, P., and Clough, P., editors, Working Notes of CLEF 2011, Amsterdam, The Netherlands.
  4. Fahlman, S. E. (2006). Marker-passing inference in the scone knowledge-base system. In the First Annual International Conference on Knowledge Science, Engineering, and Management (KSEM 2006), Guilin, China.
  5. Fan, J., Barker, K., and Porter, B. (2003). The knowledge required to interpret noun compounds. Technical Report UT-AI-TR-03-301, University of Texas at Austin.
  6. Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., and Forsyth, D. (2010). Every picture tells a story: generating sentences from images. In Proceedings of ECCV 2010, Greece.
  7. Fellbaum, C. (1998). WordNet An Electronic Lexical Database. Bradford Books.
  8. Gangemi, A., Guarino, N., Masolo, C., and Oltramari, A. (2003). Sweetening wordnet with dolce. AI Magazine, 24(3):13 - 24.
  9. Grübinger, M., Clough, P., Mller, H., and Deselaers, T. (2006). The IAPR TC-12 benchmark: A new evaluation resource for visual information systems. In International Workshop OntoImage2006 Language Resources for Content-Based Image Retrieval, held in conjuction with LREC'06, pages 13-23, Genoa, Italy.
  10. Hickl, A. and Bensley, J. (2007). A discourse commitmentbased framework for recognizing textual entailment. In ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague. ACL.
  11. Huiskes, M. J. and Lew, M. S. (2008). The mir flickr retrieval evaluation. In MIR 7808: Proceedings of the 2008 ACM International Conference on Multimedia Information Retrieval, New York, NY, USA. ACM.
  12. Leong, C. W. and Mihalcea, R. (2011). Measuring the semantic relatedness between words and images. In IWCS 7811 Proceedings of the Ninth International Conference on Computational Semantics, pages 185-194. Association for Computational Linguistics.
  13. Leong, C. W., Mihalcea, R., and Hassan, S. (2010). Text mining for automatic image tagging. In Coling 2010: Posters, pages 647-655, Beijing, China. Coling 2010 Organizing Committee.
  14. Masolo, C., Borgo, S., Gangemi, A., Guarino, N., Oltramari, A., and Schneider, L. (2003). The wonderweb library of foundational ontologies. Technical Report WonderWeb Deliverable D17, National Research Council, Institute of Cognitive Sciences and Technology (ISTC-CNR).
  15. Montazeri, N. and Hobbs, J. R. (2011). Elaborating a knowledge base for deep lexical semantics. In Bos, J. and Pulman, S., editors, Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011), pages 195-204.
  16. Müller, H., Marchand-Maillet, S., and Pun, T. (2002). The truth about corel - evaluation in image retrieval. In Lew, M. S., Sebe, N., and Eakins, J. P., editors, Lecture Notes In Computer Science, volume 2383, pages 38-49. Springer-Verlag, London.
  17. Ponte, J. M. and Croft, W. B. (1998). A language modeling approach to information retrieval. In SIGIR 1998, pages 275-281.
  18. Rashtchian, C., Young, P., Hodosh, M., and Hockenmaier, J. (2010). Collecting image annotations using amazons mechanical turk. In NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazons Mechanical Turk. Association for Computational Linguistics.
  19. Strohman, T., Metzler, D., Turtle, H., and Croft, W. B. (2005). Indri: A language model-based search engine for complex queries. In International Conference on Intelligence Analysis (ICIA) (poster), McLean, VA.
  20. Tsikrika, T., Popescu, A., and Kludas, J. (2011). Overview of the wikipedia image retrieval task at imageclef 2011. In Working Notes for the CLEF 2011 Labs and Workshop, Amsterdam, The Netherlands.
  21. Turtle, H. and Croft, W. B. (1991). Evaluation of an inference network based retrieval model. Trans. Inf. Syst., 9(3):187-222.
  22. von Ahn, L. and Dabbish, L. (2004). Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 319-326. ACM Press, Vienna, Austria.

Paper Citation

in Harvard Style

Sagae A. and Fahlman S. (2013). Knowledge Resource Development for Identifying Matching Image Descriptions . In Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2013) ISBN 978-989-8565-81-5, pages 100-108. DOI: 10.5220/0004550601000108

in Bibtex Style

author={Alicia Sagae and Scott E. Fahlman},
title={Knowledge Resource Development for Identifying Matching Image Descriptions},
booktitle={Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2013)},

in EndNote Style

JO - Proceedings of the International Conference on Knowledge Engineering and Ontology Development - Volume 1: KEOD, (IC3K 2013)
TI - Knowledge Resource Development for Identifying Matching Image Descriptions
SN - 978-989-8565-81-5
AU - Sagae A.
AU - Fahlman S.
PY - 2013
SP - 100
EP - 108
DO - 10.5220/0004550601000108