Interaction Patterns in Computer-assisted Semantic Annotation of Text - An Empirical Evaluation

Jaroslav Dytrych, Pavel Smrz

2016

Abstract

This paper examines user interface options and interaction patterns evinced in tools for computer-assisted semantic enrichment of text. It focuses on advanced annotation tasks such as hierarchical annotation of complex relations and linking entities with highly ambiguous names and explores how decisions on particular aspects of annotation interfaces influence the speed and the quality of computer-assisted human annotation processes. Reported experiments compare the 4A annotation system, designed and implemented by our team, to RDFaCE and GATE tools that all provide advanced annotation functionality. Results show that users are able to reach better consistency of event annotations in less time when using the 4A editor. A set of experiments is then conducted that employ 4A’s high flexibility and customizability to find an optimal amount of displayed information and its presentation form to reach best results in linking entities with highly ambiguous names. The last set of experiments then proves that 4A’s particular way of implementing the concept of semantic filtering speeds up event annotation processes and brings higher consistency when compared to alternative approaches.

References

  1. Bontcheva, K., Cunningham, H., Roberts, I., Roberts, A., Tablan, V., Aswani, N., and Gorrell, G. (2013). GATE Teamware: A web-based, collaborative text annotation framework. Lang. Resour. Eval., 47(4):1007- 1029.
  2. Bontcheva, K., Roberts, I., Derczynski, L., and Rout, D. (2014). The GATE Crowdsourcing Plugin: Crowdsourcing annotated corpora made easy. In Proceedings of Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 97-100. Association for Computational Linguistics.
  3. Ciccarese, P., Ocana, M., and Clark, T. (2012). Open semantic annotation of scientific publications using DOMEO. Journal of Biomedical Semantics, 3(Suppl 1).
  4. Cunningham, H., Maynard, D., Bontcheva, K., Tablan, V., Aswani, N., Roberts, I., Gorrell, G., Funk, A., Roberts, A., Damljanovic, D., Heitz, T., Greenwood, M. A., Saggion, H., Petrak, J., Li, Y., and Peters, W. (2011). Text Processing with GATE (Version 6). GATE.
  5. Grassi, M., Morbidoni, C., Nucci, M., Fonda, S., and Donato, F. D. (2013). Pundit: Creating, exploring and consuming semantic annotations. In Proceedings of the 3nd International Workshop on Semantic Digital Archives, Valletta, Malta.
  6. Handschuh, S., Staab, S., and Ciravegna, F. (2002). SCREAM - Semi-automatic CREAtion of Metadata Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web. In GómezPérez, A. and Benjamins, V., editors, Knowledge Engineering and Knowledge Management: Ontologies and the Semantic Web, volume 2473 of Lecture Notes in Computer Science, chapter 32, pages 165-184. Springer, Berlin, Heidelberg.
  7. Heese, R., Luczak-Rsch, M., Paschke, A., Oldakowski, R., and Streibel, O. (2010). One click annotation. In Proceedings of the 6th Workshop on Scripting and Development for the Semantic Web, collocated with ESWC. Ruzica Piskac, Redaktion Sun SITE, Informatik V, RWTH Aachen, Ahornstr. 55, 52056 Aachen, Germany.
  8. Hogenboom, F., Frasincar, F., Kaymak, U., and de Jong, F. (2011). An Overview of Event Extraction from Text. DeRiVE.
  9. Khalili, A., Auer, S., and Hladky, D. (2012). The RDFa Content Editor - From WYSIWYG to WYSIWYM. In Proceedings of COMPSAC 2012 - Trustworthy Software Systems for the Digital Society.
  10. Kim, J., Ohta, T., Pyysalo, S., Kano, Y., and Tsujii, J. (2009). Overview of BioNLP'09 shared task on event extraction. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing: Shared Task, pages 1-9. Association for Computational Linguistics.
  11. Maynard, D. (2008). Benchmarking textual annotation tools for the semantic web. In 6th International Conference on Language Resources and Evaluation, Marrakech, Morocco. European Language Resources Association (ELRA).
  12. Maynard, D., Dasiopoulou, S., Costache, S., Eckert, K., Stuckenschmidt, H., Dzbor, M., and Handschuh, S. (2007). Knowledge web project: Deliverable D1.2.2.1.3 - Benchmarking of annotation tools.
  13. Moro, A. and Navigli, R. (2015). SemEval-2015 Task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation, pages 288-297, Denver, Colorado.
  14. Piccinno, F. and Ferragina, P. (2014). From TagME to WAT: a new entity annotator. In Proceedings of the first international workshop on Entity recognition & disambiguation, pages 55-62. ACM.
  15. Reeve, L. and Han, H. (2005). Survey of semantic annotation platforms. In Proceedings of the 2005 ACM Symposium on Applied Computing, SAC 7805, pages 1634-1638, New York, NY, USA. ACM.
  16. R öder, M., Usbeck, R., and Ngonga Ngomo, A.-C. (2015). Developing a sustainable platform for entity annotation benchmarks. In ESWC Developers Workshop 2015. http://svn.aksw.org/papers/2015/ESWC_ GERBIL_semdev/public.pdf.
  17. Smrz, P. and Dytrych, J. (2011). Towards new scholarly communication: A case study of the 4a framework. In SePublica, volume 721 of CEUR Workshop Proceedings. Ruzica Piskac, Redaktion Sun SITE, Informatik V, RWTH Aachen, Ahornstr. 55, 52056 Aachen, Germany.
  18. Smrz, P. and Dytrych, J. (2015). Advanced features of collaborative semantic annotators - the 4a system. In Proceedings of the 28th International FLAIRS Conference, Hollywood, Florida, USA. AAAI Press.
  19. Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S., and Tsujii, J. (2012). BRAT: A web-based tool for nlp-assisted text annotation. In Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics, EACL 7812, pages 102-107, Stroudsburg, PA, USA. Association for Computational Linguistics.
  20. Surdeanu, M. and Heng, J. (2014). Overview of the English slot filling track at the TAC2014 knowledge base population evaluation. In Proceedings of the TAC-KBP 2014 Workshop.
  21. Wang, A., Hoang, C., and Kan, M.-Y. (2013). Perspectives on crowdsourcing annotations for natural language processing. Language Resources and Evaluation, 47(1):9-31.
  22. Yee, K. P. (2002). Critlink: Advanced hyperlinks enable public annotation on the web. http://zesty.ca/pubs/cscw-2002-crit.pdf.
Download


Paper Citation


in Harvard Style

Dytrych J. and Smrz P. (2016). Interaction Patterns in Computer-assisted Semantic Annotation of Text - An Empirical Evaluation . In Proceedings of the 8th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-172-4, pages 74-84. DOI: 10.5220/0005695900740084


in Bibtex Style

@conference{icaart16,
author={Jaroslav Dytrych and Pavel Smrz},
title={Interaction Patterns in Computer-assisted Semantic Annotation of Text - An Empirical Evaluation},
booktitle={Proceedings of the 8th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2016},
pages={74-84},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005695900740084},
isbn={978-989-758-172-4},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 8th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Interaction Patterns in Computer-assisted Semantic Annotation of Text - An Empirical Evaluation
SN - 978-989-758-172-4
AU - Dytrych J.
AU - Smrz P.
PY - 2016
SP - 74
EP - 84
DO - 10.5220/0005695900740084