On Continuous Top-k Similarity Joins

Da Jun Li, En Tzu Wang, Yu-Chou Tsai, Arbee L. P. Chen

Abstract

Given a similarity function and a threshold  within a range of [0, 1], a similarity join query between two sets of records returns pairs of records from the two sets, which have similarity values exceeding or equaling . Similarity joins have received much research attention since it is a fundamental operation used in a wide range of applications such as duplicate detection, data integration, and pattern recognition. Recently, a variant of similarity joins is proposed to avoid the need to set the threshold , i.e. top-k similarity joins. Since data in many applications are generated as a form of continuous data streams, in this paper, we make the first attempt to solve the problem of top-k similarity joins considering a dynamic environment involving a data stream, named continuous top-k similarity joins. Given a set of records as the query, we continuously output the top-k pairs of records, ranked by their similarity values, for the query and the most recent data, i.e. the data contained in the sliding window of a monitored data stream. Two algorithms are proposed to solve this problem. The first one extends an existing approach for static datasets to find the top-k pairs regarding the query and the newly arrived data and then keep the obtained pairs in a candidate result set. As a result, the top-k pairs can be found from the candidate result set. In the other algorithm, the records in the query are preprocessed to be indexed using a novel data structure. By this structure, the data in the monitored stream can be compared with all records in the query at one time, substantially reducing the processing time of finding the top-k results. A series of experiments are performed to evaluate the two proposed algorithms and the experiment results demonstrate that the algorithm with preprocessing outperforms the other algorithm extended from an existing approach for a static environment.

References

  1. Arasu, A., Ganti, V., and Kaushik, R., 2006. Efficient exact set-similarity joins. In Proceedings of the 32nd International Conference on Very Large Data Bases, VLDB2006. pp. 918-929.
  2. Broder, A., Glassman, S., Manasse, M., and Zweig, G., 1997 Syntactic clustering of the web. Computer Networks, vol. 29, no. 8-13, (1997) pp. 1157-1166.
  3. Bayardo, R., Ma, Y., and Srikant, R., 2007. Scaling up all pairs similarity search. The 16th International World Wide Web Conference, WWW2007, New York, NY, USA, pp. 131-140.
  4. Chowdhury, A., Frieder, O., Grossman, D., and McCabe, M., 2002. Collection statistics for fast duplicate document detection. ACM Trans. Inf. Syst., vol. 20, no. 2, (2002) pp. 171-191.
  5. Chaudhuri, S., Ganti, V., and Kaushik, R., 2006. A primitive operator for similarity joins in data cleaning. In Proceedings of the 22nd International Conference on Data Engineering, ICDE2006, Atlanta, Georgia.
  6. Cohen, W., 1998. Integration of heterogeneous databases without common domains using queries based on textual similarity. In Proceedings of the ACM Special Interest Group on Management of Data, SIGMOD1998, New York, NY, USA, pp. 201-212.
  7. Charikar, M., 2002. Similarity estimation techniques from rounding algorithms. In Proceedings of the 34th Annual ACM Symposium on Theory of Computing, STOC2002, Montreal, Quebec, Canada, pp. 380-388.
  8. Gionis, A., Indyk, P., and Motwani, R., 1999. Similarity search in high dimensions via hashing. In Proceedings of the 25th International Conference on Very Large Data Bases, VLDB1999, Edinburgh, Scotland, UK, pp. 518-529.
  9. Henzinger, M., 2006. Finding near-duplicate web pages: a large-scale evaluation of algorithms. In Proceedings of the ACM Special Interest Group on Information retrieval, SIGIR2006, New York, NY, USA. pp. 284-291.
  10. Sarawagi, S., and Bhamidipaty, A., 2002. Interactive deduplication using active learning. ACM Special Interest Group on Knowledge Discovery in Data, KDD2002, New York, NY, USA, pp. 269-278.
  11. Sarawagi, S., and Kirpal, A., 2004. Efficient set joins on similarity predicates. In Proceedings of the ACM Special Interest Group on Management of Data, SIGMOD2004, New York, NY, USA, pp. 743-754.
  12. Xiao, C., Wang, W., Lin, X., Shang, H., 2009. Top-k set similarity joins. 25th International Conference on Data Engineering. ICDE2009, Shanghai, China, pp. 916-927.
  13. Xiao, C., Wang, W., Lin, X., and Yu, J. X., 2008. Efficient similarity joins for near duplicate detection. In Proceedings of the 17th International World Wide Web Conference, WWW2008, New York, NY, USA, pp. 131-140.
Download


Paper Citation


in Harvard Style

Jun Li D., Tzu Wang E., Tsai Y. and L. P. Chen A. (2012). On Continuous Top-k Similarity Joins . In Proceedings of the International Conference on Data Technologies and Applications - Volume 1: DATA, ISBN 978-989-8565-18-1, pages 87-96. DOI: 10.5220/0003993200870096


in Bibtex Style

@conference{data12,
author={Da Jun Li and En Tzu Wang and Yu-Chou Tsai and Arbee L. P. Chen},
title={On Continuous Top-k Similarity Joins},
booktitle={Proceedings of the International Conference on Data Technologies and Applications - Volume 1: DATA,},
year={2012},
pages={87-96},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003993200870096},
isbn={978-989-8565-18-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Data Technologies and Applications - Volume 1: DATA,
TI - On Continuous Top-k Similarity Joins
SN - 978-989-8565-18-1
AU - Jun Li D.
AU - Tzu Wang E.
AU - Tsai Y.
AU - L. P. Chen A.
PY - 2012
SP - 87
EP - 96
DO - 10.5220/0003993200870096