Predicting the Stability of Large-scale Distributed Stream Processing Systems on the Cloud

Tri Minh Truong, Aaron Harwood, Richard O. Sinnott

2017

Abstract

Large-scale topology-based stream processing systems are non-trivial to build and deploy. They require understanding of the performance, cost of deployment and considerations of potential downtime. Our work considers stability as a primary characteristic of these systems. By stability, we mean that unstable systems exhibit large-spikes in latency and can drop throughput frequently or unpredictably. Such instabilities can be due to variations of workloads or underlying hardware platforms that are often difficult to predict. To understand and tackle this for large-scale stream processing systems, we apply queueing theory and simulate the results through a series of experiments on the Cloud.

References

  1. Abadi, D. J., Carney, D., C¸etintemel, U., Cherniack, M., Convey, C., Lee, S., Stonebraker, M., Tatbul, N., and Zdonik, S. (2003). Aurora: a new model and architecture for data stream management. The VLDB JournalThe International Journal on Very Large Data Bases, 12(2):120-139.
  2. Aniello, L., Baldoni, R., and Querzoni, L. (2013). Adaptive online scheduling in storm. In Proceedings of the 7th ACM international conference on Distributed eventbased systems, pages 207-218. ACM.
  3. Cardellini, V., Grassi, V., Lo Presti, F., and Nardelli, M. (2015). Distributed qos-aware scheduling in storm. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, pages 344-347. ACM.
  4. Chatzistergiou, A. and Viglas, S. D. (2014). Fast heuristics for near-optimal task allocation in data stream processing over clusters. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, pages 1579-1588. ACM.
  5. Condie, T., Conway, N., Alvaro, P., Hellerstein, J. M., Elmeleegy, K., and Sears, R. (2010). Mapreduce online. In Nsdi, volume 10, page 20.
  6. Dean, J. and Ghemawat, S. (2008). Mapreduce: Simplified data processing on large clusters. Communications of the ACM, 51(1):107-113.
  7. Eidenbenz, R. and Locher, T. (2016). for distributed stream processing. arXiv:1601.06060.
  8. Gedik, B., Schneider, S., Hirzel, M., and Wu, K.-L. (2014). Elastic scaling for data stream processing. IEEE Transactions on Parallel and Distributed Systems, 25(6):1447-1463.
  9. Heinze, T., Aniello, L., Querzoni, L., and Jerzak, Z. (2014a). Cloud-based data stream processing. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, pages 238-245. ACM.
  10. Heinze, T., Jerzak, Z., Hackenbroich, G., and Fetzer, C. (2014b). Latency-aware elastic scaling for distributed data stream processing systems. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, pages 13-22. ACM.
  11. Heinze, T., Pappalardo, V., Jerzak, Z., and Fetzer, C. (2014c). Auto-scaling techniques for elastic data stream processing. In Data Engineering Workshops (ICDEW), 2014 IEEE 30th International Conference on, pages 296-302. IEEE.
  12. Heinze, T., Roediger, L., Meister, A., Ji, Y., Jerzak, Z., and Fetzer, C. (2015). Online parameter optimization for elastic data stream processing. In Proceedings of the Sixth ACM Symposium on Cloud Computing, pages 276-287. ACM.
  13. Hu, H., Wen, Y., Chua, T.-S., and Li, X. (2014). Toward scalable systems for big data analytics: A technology tutorial. IEEE Access, 2:652-687.
  14. Jamshidi, P. and Casale, G. (2016). An uncertainty-aware approach to optimal configuration of stream processing systems. In Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), 2016 IEEE 24th International Symposium on, pages 39-48. IEEE.
  15. Kingman, J. (1961). The single server queue in heavy traffic. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 57, pages 902-904. Cambridge Univ Press.
  16. Kleppmann, M. and Kreps, J. (2015). Kafka, samza and the unix philosophy of distributed data. IEEE Data Engineering Bulletin.
  17. Krempl, G., Z?liobaite, I., BrzeziÁski, D., Hüllermeier, E., Last, M., Lemaire, V., Noack, T., Shaker, A., Sievi, S., Spiliopoulou, M., et al. (2014). Open challenges for data stream mining research. ACM SIGKDD explorations newsletter, 16(1):1-10.
  18. Kulkarni, S., Bhagat, N., Fu, M., Kedigehalli, V., Kellogg, C., Mittal, S., Patel, J. M., Ramasamy, K., and Taneja, S. (2015). Twitter heron: Stream processing at scale. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pages 239-250. ACM.
  19. Lohrmann, B., Janacik, P., and Kao, O. (2015). Elastic stream processing with latency guarantees. In Distributed Computing Systems (ICDCS), 2015 IEEE 35th International Conference on, pages 399-410. IEEE.
  20. NeCTAR (2016). The National e-Research Collaboration Tools and Resources project. https://nectar.org. au/.
  21. Neumeyer, L., Robbins, B., Nair, A., and Kesari, A. (2010). S4: Distributed stream computing platform. In 2010 IEEE International Conference on Data Mining Workshops, pages 170-177. IEEE.
  22. Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., and Seltzer, M. (2006). Network-aware operator placement for stream-processing systems. In 22nd International Conference on Data Engineering (ICDE'06), pages 49-49. IEEE.
  23. Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al. (2014). Storm@ twitter. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 147-156. ACM.
  24. Xu, L., Peng, B., and Gupta, I. (2016). Stela: Enabling stream processing systems to scale-in and scale-out on-demand. In IEEE International Conference on Cloud Engineering (IC2E).
  25. Zaharia, M., Chowdhury, M., Franklin, M. J., Shenker, S., and Stoica, I. (2010). Spark: Cluster computing with working sets. HotCloud, 10:10-10.
  26. Zikopoulos, P., Eaton, C., et al. (2011). Understanding big data: Analytics for enterprise class hadoop and streaming data. McGraw-Hill Osborne Media.
Download


Paper Citation


in Harvard Style

Truong T., Harwood A. and Sinnott R. (2017). Predicting the Stability of Large-scale Distributed Stream Processing Systems on the Cloud . In Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-243-1, pages 603-610. DOI: 10.5220/0006357606030610


in Bibtex Style

@conference{closer17,
author={Tri Minh Truong and Aaron Harwood and Richard O. Sinnott},
title={Predicting the Stability of Large-scale Distributed Stream Processing Systems on the Cloud},
booktitle={Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2017},
pages={603-610},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006357606030610},
isbn={978-989-758-243-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - Predicting the Stability of Large-scale Distributed Stream Processing Systems on the Cloud
SN - 978-989-758-243-1
AU - Truong T.
AU - Harwood A.
AU - Sinnott R.
PY - 2017
SP - 603
EP - 610
DO - 10.5220/0006357606030610