Evaluating Better Document Representation in Clustering with Varying Complexity

Stephen Bradshaw, Colm O’Riordan

2018

Abstract

Micro blogging has become a very popular activity and the posts made by users can be a valuable source of information. Classifying this content accurately can be a challenging task due to the fact that comments are typically short in nature and on their own may lack context. Reddita is a very popular microblogging site whose popularity has seen a huge and consistent increase over the years. In this paper we propose using alternative but related Reddit threads to build language models that can be used to disambiguate intend mean of terms in a post. A related thread is one which is similar in content, often consisting of the same frequently occurring terms or phrases. We posit that threads of a similar nature use similar language and that the identification of related threads can be used as a source to add context to a post, enabling more accurate classification. In this paper, graphs are used to model the frequency and co-occurrence of terms. The terms of a document are mapped to nodes, and the co-occurrence of two terms are recorded as edge weights. To show the robustness of our approach, we compare the performance in using related Reddit threads to the use of an external ontology; Wordnet. We apply a number of evaluation metrics to the clusters created and show that in every instance, the use of alternative threads to improve document representations is better than the use of Wordnet or standard augmented vector models. We apply this approach to increasingly harder environments to test the robustness of our approach. A tougher environment is one where the classifying algorithm has more than two categories to choose from when selecting the appropriate class.

Paper Citation

in Harvard Style

Bradshaw S. and O’Riordan C. (2018). Evaluating Better Document Representation in Clustering with Varying Complexity. In Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR; ISBN 978-989-758-330-8, SciTePress, pages 194-202. DOI: 10.5220/0006930901940202

in Bibtex Style

@conference{kdir18,
author={Stephen Bradshaw and Colm O’Riordan},
title={Evaluating Better Document Representation in Clustering with Varying Complexity},
booktitle={Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR},
year={2018},
pages={194-202},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006930901940202},
isbn={978-989-758-330-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2018) - Volume 1: KDIR
TI - Evaluating Better Document Representation in Clustering with Varying Complexity
SN - 978-989-758-330-8
AU - Bradshaw S.
AU - O’Riordan C.
PY - 2018
SP - 194
EP - 202
DO - 10.5220/0006930901940202
PB - SciTePress