![](bg8.png)
capable of allocating only the necessary resources
by scaling micro-services according to current work-
loads. However, this approach leaves the application’s
performances more dependent on the cloud services
they interact with.
Platform operators use benchmarks to provide
users with reliability assurances for their services.
However, the more benchmarking tools are used, the
more time-consuming their management becomes.
Therefore, in this paper, we present ISABEL, a
benchmark suite framework for Cloud-Native plat-
forms. ISABEL lessens the burden of benchmarking
multiple services by providing an extensible bench-
mark marketplace.
We describe ISABEL’s architecture and the proce-
dures of execution and registration of new benchmark
tools. A proof-of-concept implementation on Pivotal
Platform is presented, and metrics obtained by bench-
marking a service are discussed. As future work, we
intend to extend the capabilities of ISABEL to an-
alyze metrics obtained from benchmarking services,
providing a piece of more detailed performance infor-
mation for platform operators.
ACKNOWLEDGEMENTS
This work was supported by the PDTI Program,
funded by Dell Computadores do Brasil Ltda (Law
8.248 / 91).
REFERENCES
Alhamazani, K., Ranjan, R., Jayaraman, P. P., Mitra, K.,
Liu, C., Rabhi, F., Georgakopoulos, D., and Wang,
L. (2015). Cross-layer multi-cloud real-time appli-
cation qos monitoring and benchmarking as-a-service
framework. IEEE Transactions on Cloud Computing,
7(1):48–61.
Bauer, E. and Adams, R. (2012). Reliability and availability
of cloud computing. John Wiley & Sons.
Binnig, C., Kossmann, D., Kraska, T., and Loesing, S.
(2009). How is the weather tomorrow?: Towards
a benchmark for the cloud. In Proceedings of the
Second International Workshop on Testing Database
Systems, DBTest ’09, pages 9:1–9:6, New York, NY,
USA. ACM.
Chen, J., He, X., Lin, Q., Xu, Y., Zhang, H., Hao, D., Gao,
F., Xu, Z., Dang, Y., and Zhang, D. (2019). An empir-
ical investigation of incident triage for online service
systems. In 2019 IEEE/ACM 41st International Con-
ference on Software Engineering: Software Engineer-
ing in Practice (ICSE-SEIP), pages 111–120. IEEE.
Chhetri, M. B., Chichin, S., Vo, Q. B., and Kowalczyk,
R. (2013). Smart cloudbench–automated performance
benchmarking of the cloud. In 2013 IEEE Sixth Inter-
national Conference on Cloud Computing, pages 414–
421. IEEE.
Dillon, T., Wu, C., and Chang, E. (2010). Cloud computing:
Issues and challenges. In 2010 24th IEEE Interna-
tional Conference on Advanced Information Network-
ing and Applications, pages 27–33.
Ferdman, M., Adileh, A., Kocberber, O., Volos, S., Al-
isafaee, M., Jevdjic, D., Kaynak, C., Popescu, A. D.,
Ailamaki, A., and Falsafi, B. (2012). Clearing the
clouds: a study of emerging scale-out workloads on
modern hardware. Acm sigplan notices, 47(4):37–48.
Gan, Y., Zhang, Y., Cheng, D., Shetty, A., Rathi, P., Katarki,
N., Bruno, A., Hu, J., Ritchken, B., Jackson, B.,
et al. (2019). An open-source benchmark suite for mi-
croservices and their hardware-software implications
for cloud & edge systems. In Proceedings of the
Twenty-Fourth International Conference on Architec-
tural Support for Programming Languages and Oper-
ating Systems, pages 3–18.
Gannon, D., Barga, R., and Sundaresan, N. (2017). Cloud-
native applications. IEEE Cloud Computing, 4(5):16–
21.
Garg, S. K., Versteeg, S., and Buyya, R. (2011). Smicloud:
A framework for comparing and ranking cloud ser-
vices. In 2011 Fourth IEEE International Conference
on Utility and Cloud Computing, pages 210–218.
Gregg, B. (2019). BPF Performance Tools. Addison-
Wesley Professional, City.
Jin, S., Seol, J., and Maeng, S. (2013). Towards assurance
of availability in virtualized cloud system. In 2013
13th IEEE/ACM International Symposium on Cluster,
Cloud, and Grid Computing, pages 192–193.
Kasture, H. and Sanchez, D. (2016). Tailbench: a bench-
mark suite and evaluation methodology for latency-
critical applications. In 2016 IEEE International Sym-
posium on Workload Characterization (IISWC), pages
1–10. IEEE.
Kopytov, A. (2012). Sysbench manual. MySQL AB, pages
2–3.
Li, A., Yang, X., Kandula, S., and Zhang, M. (2010). Cloud-
cmp: comparing public cloud providers. In Proceed-
ings of the 10th ACM SIGCOMM conference on Inter-
net measurement, pages 1–14.
Mell, P., Grance, T., et al. (2011). The nist definition of
cloud computing.
Namiot, D. and Sneps-Sneppe, M. (2014). On micro-
services architecture. International Journal of Open
Information Technologies, 2(9):24–27.
Sfondrini, N., Motta, G., and Longo, A. (2018). Public
cloud adoption in multinational companies: A survey.
In 2018 IEEE International Conference on Services
Computing (SCC), pages 177–184.
Winn, D. C. (2017). Cloud Foundry: The Definitive Guide:
Develop, Deploy, and Scale. ” O’Reilly Media, Inc.”.
ISABEL: Infrastructure-Agnostic Benchmark Framework for Cloud-Native Platforms
489