SEMANTIC MINING OF DOCUMENTS IN A RELATIONAL DATABASE

Kunal Mukerjee, Todd Porter, Sorin Gherman

2011

Abstract

Automatically mining entities, relationships, and semantics from unstructured documents and storing these in relational tables, greatly simplifies and unifies the work flows and user experiences of database products at the Enterprise. This paper describes three linear scale, incremental, and fully automatic semantic mining algorithms that are at the foundation of the new Semantic Platform being released in the next version of SQL Server. The target workload is large (10 – 100 million) enterprise document corpuses. At these scales, anything short of linear scale and incremental is costly to deploy. These three algorithms give rise to three weighted physical indexes: Tag Index (top keywords in each document); Document Similarity Index (top closely related documents given any document); and Phrase Similarity Index (top semantically related phrases, given any phrase), which are then query-able through the SQL interface. The need for specifically creating these three indexes was motivated by observing typical stages of document research, and gap analysis, given current tools and technology at the Enterprise. We describe the mining algorithms and architecture, and outline some compelling user experiences that are enabled by these indexes.

References

  1. Administering Full Text Search (http://msdn.microsoft. com/en-us/library/ms142557.aspx).
  2. Baeza-Yates, R., Ribeiro-Neto, B., Modern Information Retrieval, Addison-Wesley, 1999.
  3. Blei, D. M., Ng, A. Y., Jordan, M., Latent Dirichlet allocation, Journal of Machine Learning Research No. 3, pp. 993 - 1022, 2003.
  4. Cohen, J. D., Highlights: Language- and domain independent automatic indexing terms for abstracting, Journal of the American Society of Information Science, Volume 46, Issue 3, pp. 162-174, April 1995.
  5. Damashek, M., Gauging similarity with N-grams: Language-independent categorization of text, Science 267, Feb. 1995.
  6. Deerwester, S., et al., Improving Information Retrieval with Latent Semantic Indexing, Proceedings of the 51st Annual Meeting of the American Society for Information Science 25, 1988, pp. 36-40.
  7. Full Text Search Overview (http://msdn.microsoft.com/enus/library/ms142571.aspx).
  8. Gabrilovich, E., Markovitch, S., Computing semantic relatedness using Wikipedia-based explicit semantic analysis, in Proceedings of the 20th International Joint Conference on Artificial Intelligence, 2007.
  9. Hammouda, K., M., Kamel, M., S., Document Similarity using a Phrase Indexing Graph Model, Journal of Knowledge and Information Systems, Vol. 6, Issue 6, November 2004.
  10. Hofmann, T., “Probabilistic Latent Semantic Indexing”, Proc. SIGIR, 1999.
  11. Jiang, X., Hu, Y., Li, H., A Ranking Approach to Keyphrase Extraction, Microsoft Research Technical Report, 2009.
  12. McNabb, K., Moore, C., and Levitt, D., Open Text Leads ECM Suite Pure Plays, The Forrester Wave Vendor Summary, Q4 2007.
  13. Salton, G., Wong, A., Yang, C. S., “A Vector Space Model for Automatic Indexing”, Communications of the ACM, vol. 18, nr. 11, pages 613-620, 1975.
  14. Silberschatz, A., Stonebraker, M., and Ullman, J., Database Research: Achievements and Opportunities into the 21st Century. Technical Report, Stanford, 1996.
  15. Tan, P-N., Steinbach, M., Kumar, V., Introduction to Data Mining, 2005
  16. Witten, I. H., Paynter, G. W., Frank, E., Gutwin, C., and Nevill-Manning, C., G., KEA: Practical automatic keyphrase extraction, Proc. DL 7899, pp. 254-256.
  17. Zamir, O., Etzioni, O., Web Document Clustering: A Feasibility Demonstration, in Proc. ACM SIGIR'98, 1998.
Download


Paper Citation


in Harvard Style

Mukerjee K., Porter T. and Gherman S. (2011). SEMANTIC MINING OF DOCUMENTS IN A RELATIONAL DATABASE . In Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2011) ISBN 978-989-8425-79-9, pages 138-150. DOI: 10.5220/0003631401460158


in Bibtex Style

@conference{kdir11,
author={Kunal Mukerjee and Todd Porter and Sorin Gherman},
title={SEMANTIC MINING OF DOCUMENTS IN A RELATIONAL DATABASE},
booktitle={Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2011)},
year={2011},
pages={138-150},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003631401460158},
isbn={978-989-8425-79-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Knowledge Discovery and Information Retrieval - Volume 1: KDIR, (IC3K 2011)
TI - SEMANTIC MINING OF DOCUMENTS IN A RELATIONAL DATABASE
SN - 978-989-8425-79-9
AU - Mukerjee K.
AU - Porter T.
AU - Gherman S.
PY - 2011
SP - 138
EP - 150
DO - 10.5220/0003631401460158