Continuous Queries over Distributed Streams of Heterogeneous Monitoring Data in Cloud Datacenters

Daniel Tovarňák, Tomáš Pitner

2014

Abstract

The use of stream processing for state monitoring of distributed infrastructures has been advocated by some in order to overcome the issues of traditional monitoring solutions when tasked with complex continuous queries. However, in the domain of behavior monitoring the situation gets more complicated. It is mainly because of the low-quality source of behavior-related monitoring information (natural language computer logs). Existing approaches prevalently rely on indexing and real-time data-mining of the behavior-related data rather than on using event/stream processing techniques and the many corresponding benefits. The goal of this paper is to present a general notion of Distributed Event-Driven Monitoring Architecture that will enable an easy definition of expressive continuous queries over many distributed and heterogeneous streams of behavior-related (and state-related) monitoring data.

References

  1. Aceto, G., Botta, A., de Donato, W., and Pescapè, A. (2013). Cloud monitoring: A survey. Computer Networks, 57(9).
  2. Balis, B., Dyk, G., and Bubak, M. (2012). On-line grid monitoring based on distributed query processing. In Wyrzykowski, R., Dongarra, J., Karczewski, K., and Wasniewski, J., editors, Parallel Processing and Applied Mathematics, volume 7204 of Lecture Notes in Computer Science. Springer Berlin Heidelberg.
  3. Balis, B., Kowalewski, B., and Bubak, M. (2011). Real-time grid monitoring based on complex event processing. Future Generation Computer Systems, 27(8).
  4. Boulon, J., Konwinski, A., Qi, R., Rabkin, A., Yang, E., and Yang, M. (2008). Chukwa, a large-scale monitoring system. In Proceedings of CCA, volume 8.
  5. Brunette, G. and Mogull, R. (2009). Security guidance for critical areas of focus in cloud computing v2.1. Cloud Security Alliance, (December):1-76.
  6. Clayman, S., Galis, A., and Mamatas, L. (2010). Monitoring virtual networks with lattice. In Network Operations and Management Symposium Workshops (NOMS Wksps), 2010 IEEE/IFIP.
  7. Cre¸tu-Ciocaˆrlie, G. F., Budiu, M., and Goldszmidt, M. (2008). Hunting for problems with artemis. In Proceedings of the First USENIX conference on Analysis of system logs, WASL'08, Berkeley, CA, USA. USENIX Association.
  8. De Chaves, S., Uriarte, R., and Westphall, C. (2011). Toward an architecture for monitoring private clouds. Communications Magazine, IEEE, 49(12).
  9. Etzion, O. and Niblett, P. (2010). Event Processing in Action. Manning Publications Co., Greenwich, CT, USA, 1st edition.
  10. Gerhards, R. (2009). The Syslog Protocol. http://tools.ietf.org/html/rfc5424.
  11. Hasselmeyer, P. and d'Heureuse, N. (2010). Towards holistic multi-tenant monitoring for virtual data centers. In Network Operations and Management Symposium Workshops (NOMS Wksps), 2010 IEEE/IFIP.
  12. Hohpe, G. and Woolf, B. (2004). Enterprise integration patterns: Designing, building, and deploying messaging solutions. Addison-Wesley Professional.
  13. ISO/IEC 25010:2011 (2011). Systems and software engineering - Systems and software Quality Requirements and
  14. Kreps, J., Narkhede, N., and Rao, J. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of 6th International Workshop on Networking Meets Databases (NetDB), Athens, Greece.
  15. Lu, X., Yin, J., Li, Y., Deng, S., and Zhu, M. (2012). An efficient data dissemination approach for cloud monitoring. In Liu, C., Ludwig, H., Toumani, F., and Yu, Q., editors, Service-Oriented Computing, volume 7636 of LNCS. Springer Berlin Heidelberg.
  16. Mansouri-Samani, M. (1995). Monitoring of distributed systems. PhD thesis, Imperial College London (University of London).
  17. Massie, M. L., Chun, B. N., and Culler, D. E. (2004). The ganglia distributed monitoring system: design, implementation, and experience. Parallel Computing, 30(7).
  18. Nagappan, M. and Vouk, M. (2010). Abstracting log lines to log event types for mining software system logs. In Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on.
  19. Narayanan, K., Bose, S. K., and Rao, S. (2011). Towards 'integrated' monitoring and management of datacenters using complex event processing techniques. In Proceedings of the Fourth Annual ACM Bangalore Conference, COMPUTE 7811, NY, NY, USA. ACM.
  20. Object Management Group (2007). Data Distribution Service for Real-time Systems, Version 1.2. Technical report.
  21. Oliner, A. and Stearley, J. (2007). What supercomputers say: A study of five system logs. In Dependable Systems and Networks, 2007. DSN 7807. 37th Annual IEEE/IFIP International Conference on.
  22. Rabkin, A. and Katz, R. (2010). Chukwa: a system for reliable large-scale log collection. In Proceedings of the 24th international conference on Large installation system administration, LISA'10, Berkeley, CA, USA. USENIX Association.
  23. Rackl, G. (2001). Monitoring and Managing Heterogeneous Middleware. PhD thesis, Technische Universität München, Universitätsbibliothek.
  24. Spring, J. (2011). Monitoring Cloud Computing by Layer, Part 1, Part2. Security & Privacy, IEEE, 9(2).
  25. Teixeira, P. H. d. S., Clemente, R. G., Kaiser, R. A., and Vieira, Jr., D. A. (2010). Holmes: an event-driven solution to monitor data centers through continuous queries and machine learning. In Proceedings of the Fourth ACM International Conference on Distributed Event-Based Systems, DEBS 7810, NY, NY, USA. ACM.
  26. Tovar n?ák, D. and Pitner, T. (2012). Towards Multi-Tenant and Interoperable Monitoring of Virtual Machines in Cloud. SYNASC 2012, MICAS Workshop.
  27. Zanikolas, S. and Sakellariou, R. (2005). A taxonomy of grid monitoring systems. Future Generation Computer Systems, 21(1).
Download


Paper Citation


in Harvard Style

Tovarňák D. and Pitner T. (2014). Continuous Queries over Distributed Streams of Heterogeneous Monitoring Data in Cloud Datacenters . In Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014) ISBN 978-989-758-036-9, pages 470-481. DOI: 10.5220/0005095504700481


in Bibtex Style

@conference{icsoft-ea14,
author={Daniel Tovarňák and Tomáš Pitner},
title={Continuous Queries over Distributed Streams of Heterogeneous Monitoring Data in Cloud Datacenters},
booktitle={Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014)},
year={2014},
pages={470-481},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005095504700481},
isbn={978-989-758-036-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014)
TI - Continuous Queries over Distributed Streams of Heterogeneous Monitoring Data in Cloud Datacenters
SN - 978-989-758-036-9
AU - Tovarňák D.
AU - Pitner T.
PY - 2014
SP - 470
EP - 481
DO - 10.5220/0005095504700481