Internal Network Monitoring and Anomaly Detection through Host Clustering

W. J. B. Beukema, T. Attema, H. A. Schotanus


Internal network traffic is an undervalued source of information for detecting targeted attacks. Whereas most systems focus on the external border of the network, we observe that targeted attacks campaigns often involve internal network activity. To this end, we have developed techniques capable of detecting anomalous internal network behaviour. As a second contribution we propose an additional step in the model-based anomaly detection involving host clustering. Through host clustering, individual hosts are grouped together on the basis of their internal network behaviour. We argue that a behavioural model for each cluster, compared to a model for each host or a single model for all hosts, performs better in terms of detecting potentially malicious behaviour. We show that by applying this concept to internal network traffic, the detection performance for identifying malicious flows and hosts increases.


  1. Akoglu, L., Tong, H., and Koutra, D. (2015). Graph based anomaly detection and description: a survey. Data Mining and Knowledge Discovery, 29(3):626-688.
  2. Bencsáth, B., Pék, G., Buttyán, L., and Félegyházi, M. (2012). Duqu: Analysis, detection, and lessons learned. In ACM European Workshop on System Security (EuroSec), volume 2012.
  3. Beukema, W. J. B. (2016). Enhancing network intrusion detection through host clustering. Master's thesis, University of Twente.
  4. Bordes, A., Ertekin, S., Weston, J., and Bottou, L. (2005). Fast kernel classifiers with online and active learning. Journal of Machine Learning Research, 6(Sep):1579- 1619.
  5. Boser, B. E., Guyon, I. M., and Vapnik, V. N. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 144-152. ACM.
  6. Byrne, M. D. (2013). How many times should a stochastic model be run - An approach based on confidence intervals. In Proceedings of the 12th International conference on cognitive modeling, Ottawa.
  7. Claise, B., Quittek, J., Meyer, J., Bryant, S., and Aitken, P. (2015). Information Model for IP Flow Information Export. doi: rfc510210.17487/rfc5102.
  8. Comaniciu, D. and Meer, P. (2002). Mean Shift: A Robust Approach Toward Feature Space Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):603-619.
  9. Dell SecureWorks (2013). Advanced persistent threat analysis. Accessed on 21/01/2016.
  10. Denning, D. E. (1987). An intrusion-detection model. IEEE Transactions on Software Engineering, 13(2):222- 232.
  11. Ehrlich, W. K., Karasaridis, A., Hoeflin, D. A., and Liu, D. (2010). Detection of spam hosts and spam bots using network flow traffic modeling. In LEET.
  12. Eskin, E. (2000). Anomaly detection over noisy data using learned probability distributions. In In Proceedings of the International Conference on Machine Learning. Citeseer.
  13. Eskin, E., Arnold, A., Prerau, M., Portnoy, L., and Stolfo, S. (2002). A geometric framework for unsupervised anomaly detection. In Applications of data mining in computer security, pages 77-101. Springer.
  14. Eskin, E., Lee, W., and Stolfo, S. J. (2001). Modeling system calls for intrusion detection with dynamic window sizes. In DARPA Information Survivability Conference & Exposition II, 2001. DISCEX'01. Proceedings, volume 1, pages 165-175. IEEE.
  15. Fortunato, S. (2010). Community detection in graphs. Physics Reports, 486(3):75-174.
  16. Harris, D. and Harris, S. (2012). Digital design and computer architecture. Elsevier.
  17. Holland, P. W., Laskey, K. B., and Leinhardt, S. (1983). Stochastic blockmodels: First steps. Social Networks, 5(2):109 - 137.
  18. Hutchins, E. M., Cloppert, M. J., and Amin, R. M. (2011). Intelligence-driven computer network defense informed by analysis of adversary campaigns and intrusion kill chains. Leading Issues in Information Warfare & Security Research, 1:80.
  19. Jain, A. K. (2010). Data clustering: 50 years beyond kmeans. Pattern Recognition Letters, 31(8):651-666.
  20. Jain, A. K., Topchy, A., Law, M. H. C., and Buhmann, J. M. (2004). Landscape of Clustering Algorithms. In Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 1 - Volume 01, ICPR 7804, pages 260-263, Washington, DC, USA. IEEE Computer Society.
  21. Kaspersky (2015). Carbanak APT: The Great Bank Robbery. Accessed on 18/2/2016.
  22. Langner, R. (2011). Stuxnet: Dissecting a cyberwarfare weapon. IEEE Security and Privacy, 9(3):49-51.
  23. Lazarevic, A., Kumar, V., and Srivastava, J. (2005). Intrusion detection: A survey. In Kumar, V., Srivastava, J., and Lazarevic, A., editors, Managing Cyber Threats: Issues, Approaches, and Challenges, pages 19-78. Springer US, Boston, MA.
  24. Li, Y., Wang, J.-L., Tian, Z.-H., Lu, T.-B., and Young, C. (2009). Building lightweight intrusion detection system using wrapper-based feature selection mechanisms. Computers & Security, 28(6):466 - 475.
  25. MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1(14), pages 281- 297. Oakland, CA, USA.
  26. Research and Markets (2015). Advanced persistent threat protection market - global forecast to 2020.
  27. Roy, D. B. and Chaki, R. (2014). State of the art analysis of network traffic anomaly detection. In Applications and Innovations in Mobile Computing (AIMoC), 2014, pages 186-192. IEEE.
  28. Sabahi, F. and Movaghar, A. (2008). Intrusion Detection: A Survey. In Systems and Networks Communications, 2008. ICSNC 7808. 3rd International Conference on, pages 23-26.
  29. Scarfone, K. A. and Mell, P. M. (2007). SP 800-94. Guide to Intrusion Detection and Prevention Systems (IDPS). Technical report, National Institute of Standards & Technology, Gaithersburg, MD, United States.
  30. Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., and Williamson, R. C. (2001). Estimating the support of a high-dimensional distribution. Neural Computation, 13(7):1443-1471.
  31. Shiravi, A., Shiravi, H., Tavallaee, M., and Ghorbani, A. A. (2012). Toward developing a systematic approach to generate benchmark datasets for intrusion detection. Computers & Security, 31(3):357 - 374.
  32. Virvilis, N. and Gritzalis, D. (2013). The Big Four - What We Did Wrong in Advanced Persistent Threat Detection? In Availability, Reliability and Security (ARES), 2013 Eighth International Conference on, pages 248- 254.
  33. Wei, S., Mirkovic, J., and Kissel, E. (2006). Profiling and clustering internet hosts. DMIN, 6:269-75.
  34. Xiao, H., Gao, J., Turaga, D. S., Vu, L. H., and Biem, A. (2015). Temporal multi-view inconsistency detection for network traffic analysis. InProceedings of the 24th International Conference on World Wide Web, pages 455-465. ACM.
  35. Xu, K., Wang, F., and Gu, L. (2011). Network-aware behavior clustering of internet end hosts. In INFOCOM, 2011 Proceedings IEEE, pages 2078-2086. IEEE.
  36. Xu, R., Wunsch, D., et al. (2005). Survey of Clustering Algorithms. Neural Networks, IEEE Transactions on, 16(3):645-678.
  37. Zhang, M., Xu, B., and Gong, J. (2015). An Anomaly Detection Model Based on One-Class SVM to Detect Network Intrusions. In 2015 11th International Conference on Mobile Ad-hoc and Sensor Networks (MSN), pages 102-107.

Paper Citation

in Harvard Style

Beukema W., Attema T. and Schotanus H. (2017). Internal Network Monitoring and Anomaly Detection through Host Clustering . In Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ForSE, (ICISSP 2017) ISBN 978-989-758-209-7, pages 694-703. DOI: 10.5220/0006288606940703

in Bibtex Style

author={W. J. B. Beukema and T. Attema and H. A. Schotanus},
title={Internal Network Monitoring and Anomaly Detection through Host Clustering},
booktitle={Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ForSE, (ICISSP 2017)},

in EndNote Style

JO - Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ForSE, (ICISSP 2017)
TI - Internal Network Monitoring and Anomaly Detection through Host Clustering
SN - 978-989-758-209-7
AU - Beukema W.
AU - Attema T.
AU - Schotanus H.
PY - 2017
SP - 694
EP - 703
DO - 10.5220/0006288606940703