A Wavelet-inspired Anomaly Detection Framework for Cloud Platforms

David O'Shea, Vincent C. Emeakaroha, John Pendlebury, Neil Cafferkey, John P. Morrison, Theo Lynn

2016

Abstract

Anomaly detection in Cloud service provisioning platforms is of significant importance, as the presence of anomalies indicates a deviation from normal behaviour, and in turn places the reliability of the distributed Cloud network into question. Existing solutions lack a multi-level approach to anomaly detection in Clouds. This paper presents a wavelet-inspired anomaly detection framework for detecting anomalous behaviours across Cloud layers. It records the evolution of multiple metrics and extracts a two-dimensional spectrogram representing a monitored system’s behaviour. Over two weeks of historical monitoring data were used to train the system to identify healthy behaviour. Anomalies are then characterised as deviations from this expected behaviour. The training technique as well as the pre-processing techniques are highly configurable. Based on a Cloud service deployment use case scenario, the effectiveness of the framework was evaluated by randomly injecting anomalies into the recorded metric data and performing comparison using the resulting spectrograms.

References

  1. Agarwal, S., Mozafari, B., Panda, A., Milner, H., Madden, S., and Stoica, I. (2013). Blinkdb: queries with bounded errors and bounded response times on very large data. In Proceedings of the 8th ACM European Conference on Computer Systems, pages 29-42. ACM.
  2. Agarwala, S., Alegre, F., Schwan, K., and Mehalingham, J. (2007). E2eprof: Automated end-to-end performance management for enterprise systems. In Dependable Systems and Networks, 2007. DSN 7807. 37th Annual IEEE/IFIP International Conference on, pages 749- 758.
  3. Albanese, D., Visintainer, R., Merler, S., Riccadonna, S., Jurman, G., and Furlanello, C. (2012). mlpy: Machine learning python. http://mlpy.sourceforge.net/ Accessed on 22/02/2016.
  4. Apache Software Foundation (2016). Apache JMeter. http://jmeter.apache.org/ Accessed on 06/01/2016.
  5. Bahl, P., Chandra, R., Greenberg, A., Kandula, S., Maltz, D., and Zhang, M. (2007). Towards highly reliable enterprise network services via inference of multi-level dependencies. In SIGCOMM. Association for Computing Machinery, Inc.
  6. Bakhtazad, A., Palazoglu, A., and Romagnoli, J. (2000). Detection and classification of abnormal process situations using multidimensional wavelet domain hidden Markov trees. Computers & Chemical Engineering, 24(2):769-775.
  7. Buzen, J. P. and Shum, A. W. (1995). Masf-multivariate adaptive statistical filtering. InInt. CMG Conference, pages 1-10.
  8. Chandola, V., Banerjee, A., and Kumar, V. (2009). Anomaly detection: A survey. ACM Comput. Surv., 41(3):15:1-15:58.
  9. Doelitzscher, F., Knahl, M., Reich, C., and Clarke, N. (2013). Anomaly detection in IaaS clouds. In Cloud Computing Technology and Science (CloudCom), 2013 IEEE 5th International Conference on, pages 387-394.
  10. Emeakaroha, V. C., Brandic, I., Maurer, M., and Dustdar, S. (2010). Low level metrics to high level slas - lom2his framework: Bridging the gap between monitored metrics and sla parameters in cloud environments. In 2010 International Conference on High Performance Computing and Simulation (HPCS), pages 48 -54.
  11. Emeakaroha, V. C., Netto, M. A. S., Calheiros, R. N., Brandic, I., Buyya, R., and De Rose, C. A. F. (2012). Towards autonomic detection of sla violations in cloud infrastructures. Future Gener. Comput. Syst., 28(7):1017-1029.
  12. Fatema, K., Emeakaroha, V. C., Healy, P. D., Morrison, J. P., and Lynn, T. (2014). A survey of cloud monitoring tools: Taxanomy, capabilities and objectives. Journal of Parallel and Distributed Computing, 74:2918- 2933.
  13. Gander, M., Felderer, M., Katt, B., Tolbaru, A., Breu, R., and Moschitti, A. (2013). Anomaly detection in the cloud: Detecting security incidents via machine learning. In Trustworthy Eternal Systems via Evolving Software, Data and Knowledge, pages 103-116. Springer.
  14. Guan, Q. and Fu, S. (2013a). Adaptive anomaly identification by exploring metric subspace in cloud computing infrastructures. In Reliable Distributed Systems (SRDS), 2013 IEEE 32nd International Symposium on, pages 205-214.
  15. Guan, Q. and Fu, S. (2013b). Wavelet-based multi-scale anomaly identification in cloud computing systems. In Global Communications Conference (GLOBECOM), 2013 IEEE, pages 1379-1384.
  16. Guan, Q., Fu, S., DeBardeleben, N., and Blanchard, S. (2013). Exploring time and frequency domains for accurate and automated anomaly detection in cloud computing systems. In Dependable Computing (PRDC), 2013 IEEE 19th Pacific Rim International Symposium on, pages 196-205. IEEE.
  17. Gul, I. and Hussain, M. (2011). Distributed cloud intrusion detection model. International Journal of Advanced Science and Technology, 34:71-82.
  18. Hodge, V. J. and Austin, J. (2004). A survey of outlier detection methodologies. Artificial Intelligence Review , 22(2):85-126.
  19. Ibidunmoye, O., Hernández-Rodriguez, F., and Elmroth, E. (2015). Performance anomaly detection and bottleneck identification. ACM Comput. Surv., 48(1):4:1- 4:35.
  20. Lin, M., Yao, Z., Gao, F., and Li, Y. (2015). Toward anomaly detection in iaas cloud computing platforms. International Journal of Security and Its Applications, 9(12):175 - 188.
  21. Liu, A., Chen, J. X., and Wechsler, H. (2015). Real-time timing channel detection in an software-defined networking virtual environment. Intelligent Information Management, 7(06):283.
  22. Mi, H., Wang, H., Yin, G., Cai, H., Zhou, Q., Sun, T., and Zhou, Y. (2011a). Magnifier: Online detection of performance problems in large-scale cloud computing systems. In Services Computing (SCC), 2011 IEEE International Conference on, pages 418-425.
  23. Mi, H., Wang, H., Yin, G., Cai, H., Zhou, Q., Sun, T., and Zhou, Y. (2011b). Magnifier: Online detection of performance problems in large-scale cloud computing systems. In Services Computing (SCC), 2011 IEEE International Conference on, pages 418-425.
  24. Penn, B. S. (2005). Using self-organizing maps to visualize high-dimensional data. Computers & Geosciences, 31(5):531 - 544.
  25. Sha, W., Zhu, Y., Chen, M., and Huang, T. (2015). Statistical learning for anomaly detection in cloud server systems: A multi-order markov chain framework. Cloud Computing, IEEE Transactions on.
  26. Song, X., Wu, M., Jermaine, C., and Ranka, S. (2007). Conditional anomaly detection. IEEE Trans. on Knowl. and Data Eng., 19(5):631-645.
  27. Videla, A. and Williams, J. J. (2012). RabbitMQ in Action: Distributed Messaging for Everyone. Manning Publications Company.
  28. Wang, C., Talwar, V., Schwan, K., and Ranganathan, P. (2010). Online detection of utility cloud anomalies using metric distributions. In Network Operations and Management Symposium (NOMS), 2010 IEEE, pages 96-103.
  29. Wang, C., Viswanathan, K., Choudur, L., Talwar, V., Satterfield, W., and Schwan, K. (2011). Statistical techniques for online anomaly detection in data centers. In Integrated Network Management (IM), 2011 IFIP/IEEE International Symposium on, pages 385- 392.
  30. Welford, B. P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics, 4(3):419-420.
  31. Zhang, Z., Wang, Y., and Wang, K. (2013). Fault diagnosis and prognosis using wavelet packet decomposition, Fourier transform and artificial neural network. J. Intell. Manuf., 24(6):1213-1227.
Download


Paper Citation


in Harvard Style

O'Shea D., Emeakaroha V., Pendlebury J., Cafferkey N., Morrison J. and Lynn T. (2016). A Wavelet-inspired Anomaly Detection Framework for Cloud Platforms . In Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-182-3, pages 106-117. DOI: 10.5220/0005913701060117


in Bibtex Style

@conference{closer16,
author={David O'Shea and Vincent C. Emeakaroha and John Pendlebury and Neil Cafferkey and John P. Morrison and Theo Lynn},
title={A Wavelet-inspired Anomaly Detection Framework for Cloud Platforms},
booktitle={Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2016},
pages={106-117},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005913701060117},
isbn={978-989-758-182-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - A Wavelet-inspired Anomaly Detection Framework for Cloud Platforms
SN - 978-989-758-182-3
AU - O'Shea D.
AU - Emeakaroha V.
AU - Pendlebury J.
AU - Cafferkey N.
AU - Morrison J.
AU - Lynn T.
PY - 2016
SP - 106
EP - 117
DO - 10.5220/0005913701060117