An Automatic Tool for Benchmark Testing of Cloud Applications

Valentina Casola, Alessandra De Benedictis, Massimiliano Rak, Umberto Villano

2017

Abstract

The performance testing of cloud applications is a challenging research topic, due to the multiplicity of different possibilities to allocate application services to Cloud Service Providers (CSPs). Currently available benchmarks mainly focus on evaluating specific services or infrastructural resources offered by different CSPs, but are not always useful to evaluate complete cloud applications and to discover performance bugs. This paper proposes a methodology to define an evaluation performance process, particularly suited for cloud applications, and an automatic procedure to set up and to execute benchmark tests. The methodology is based on the evaluations of two performance indexes, and is validated by presenting a complete case study application, developed within the FP7-EU-SPECS project. The analysis of the measurement results, produced automatically, can help the developer to discover possible bottlenecks and to take actions to improve both the usability and the performance of a cloud application.

References

  1. Andreieux, A. (2007). Web services agreement specification. https://www.ogf.org/documents/GFD.107.pdf.
  2. Aversano, G., Rak, M., and Villano, U. (2013). The mOSAIC benchmarking framework: Development and execution of custom cloud benchmarks. Scalable Computing: Practice and Experience, 14(1).
  3. Binnig, C., Kossmann, D., Kraska, T., and Loesing, S. (2009). How is the weather tomorrow? Proceedings of the Second International Workshop on Testing Database Systems - DBTest 7809.
  4. Casola, V., A. De Benedictis, Rak, M., and Villano, U. (2014). Preliminary Design of a Platform-as-aService to Provide Security in Cloud. In Proceedings of CLOSER 2014, Barcelona, Spain, April 3-5, 2014, pages 752-757.
  5. Casola, V., De Benedictis, A., Rak, M., and Villano, U. (2015). SLA-based secure cloud application development: The SPECS framework. In Proceedings of SYNASC 2015, pages 337-344.
  6. Cuomo, A., Rak, M., and Villano, U. (2015). Performance prediction of cloud applications through benchmarking and simulation. International Journal of Computational Science and Engineering, 11(1):46-55.
  7. Dejun, J., Pierre, G., and Chi, C.-H. (2009). EC2 performance analysis for resource provisioning of serviceoriented applications. In Proceedings of the 2009 Int. Conf. on Service-oriented Computing, pages 197-207.
  8. ETSI (2013). Cloud standards coordination. Technical report.
  9. Gartner (2017). Cloud Harmony Web Site. https://cloudharmony.com/.
  10. Gatling Corp (2017). Gatling documentation. http://gatling.io/docs/2.2.3/.
  11. Haak, S. and Menzel, M. (2011). Autonomic benchmarking for cloud infrastructures. Proceedings of the 1st ACM/IEEE workshop on Autonomic computing in economics - ACE 7811, pages 27-32.
  12. International Organization for Standardization (2014). ISO/IEC 17788:2014. Information TechnologyCloud computing-Overview and vocabulary.
  13. Kunde, S. and Mukherjee, T. (2015). Workload characterization model for optimal resource allocation in cloud middleware. Proceedings of the 30th Annual ACM Symposium on Applied Computing - SAC 7815, pages 442-447.
  14. Li, A., Yang, X., Kandula, S., and Zhang, M. (2010). CloudCmp: Comparing Public Cloud Providers. In Proceedings of the 10th annual conference on Internet measurement - IMC 7810, pages 1-14.
  15. Li, A., Zong, X., Kandula, S., Yang, X., and Zhang, M. (2011). Cloudprophet: towards application performance prediction in cloud. In Proceedings of the ACM SIGCOMM 2011 conference, pages 426-427.
  16. Liew, S. H. and Su, Y.-Y. (2012). Cloudguide: Helping users estimate cloud deployment cost and performance for legacy web applications. In Cloud Computing Technology and Science (CloudCom), 2012 IEEE 4th Int. Conf. on, pages 90-98.
  17. Mell, P. M. and Grance, T. (2011). Sp 800-145. the nist definition of cloud computing. Technical report, Gaithersburg, MD, United States.
  18. Rak, M., Suri, N., Luna, J., Petcu, D., Casola, V., and Villano, U. (2013). Security as a service using an SLA-based approach via SPECS. In Cloud Computing Technology and Science (CloudCom), 2013 IEEE 5th Int. Conf. on, volume 2, pages 1-6.
  19. Rak, M., Turtur, M., and Villano, U. (2015). Early prediction of the cost of HPC application execution in the cloud. In Proceedings of SYNASC 2014, pages 409- 416.
  20. Scheuner, J., Cito, J., Leitner, P., and Gall, H. (2015). Cloud workbench: Benchmarking IaaS providers based on Infrastructure-as-Code. In Proc. of the 24th Int. Conf. on World Wide Web, pages 239-242.
  21. Scheuner, J., Leitner, P., Cito, J., and Gall, H. (2014). Cloud work bench-infrastructure-as-code based cloud benchmarking. In Cloud Computing Technology and Science (CloudCom), 2014 IEEE 6th International Conference on, pages 246-253.
  22. Sharma, U., Shenoy, P., Sahu, S., and Shaikh, A. (2011). Kingfisher: Cost-aware elasticity in the cloud. In 2011 Proceedings IEEE INFOCOM, pages 206-210.
  23. SPECS Consortium (2017a). SPECS Performance tests. https://bitbucket.org/specs-team/specs-performancebenchmark.
  24. SPECS Consortium (2017b). The SPECS project web site. http://specs-project.eu/.
  25. SPECS Consortium (2017c). The SPECS Secure Web Container Application - Bitbucket Repository. https://bitbucket.org/specs-team/specs-appwebcontainer-rev2.
  26. SPECS Consortium (2017d). The SPECS Secure Web Container Application Description. http://www.specsproject.eu/solutions-portofolio/secure-webcontainer/.
  27. Uhlir, V., Tomanek, O., and Kencl, L. (2016). Latencybased benchmarking of cloud service providers. In Proceedings of the 9th Int. Conf, on Utility and Cloud Computing, UCC 7816, pages 263-268.
Download


Paper Citation


in Harvard Style

Casola V., De Benedictis A., Rak M. and Villano U. (2017). An Automatic Tool for Benchmark Testing of Cloud Applications . In Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER, ISBN 978-989-758-243-1, pages 729-736. DOI: 10.5220/0006379507290736


in Bibtex Style

@conference{closer17,
author={Valentina Casola and Alessandra De Benedictis and Massimiliano Rak and Umberto Villano},
title={An Automatic Tool for Benchmark Testing of Cloud Applications},
booktitle={Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,},
year={2017},
pages={729-736},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006379507290736},
isbn={978-989-758-243-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Conference on Cloud Computing and Services Science - Volume 1: CLOSER,
TI - An Automatic Tool for Benchmark Testing of Cloud Applications
SN - 978-989-758-243-1
AU - Casola V.
AU - De Benedictis A.
AU - Rak M.
AU - Villano U.
PY - 2017
SP - 729
EP - 736
DO - 10.5220/0006379507290736