environment. This is often called the “DevOps
Paradox” (Edwards, 2015): “Going faster brings
higher quality, lower costs, and better outcomes”.
Organizations such as Etsy, Netflix, Facebook,
Amazon, Twitter and Google, but also ING, or
Target, by applying the DevOps philosophy, they
have been able to achieve levels of performance that
were unthinkable even five years ago. DevOps
pivots around three axes (UpGuard, 2016):
processes, people and technology. From the people
perspective, DevOps symbolizes a cultural change
where collaboration and cooperation are key pillars,
and this often results in an increased understanding
to prioritize requests that the business needs. From
the processes perspective, DevOps advocates for a
more agile change processes, with an increased rate
of acceptance for new features, improved quality in
software developments, a decrease in number of
incidents per release and an increased time to market
and velocity to pass from development to
production. Finally, from the technology point of
view, DevOps results in an application with a
reduced number of defects and therefore with more
quality, and in an increased deployment of features.
The authors have analyzed several DevOps
solutions existing on the market and in open source
communities, such as The Eclipse Cloud
Development project (Eclipse Cloud Development;,
2015), IBM Bluemix DevOps Services (IBM
BlueMix, 2016), or Puppet+MCollective (Puppet
labs, 2013) and Terraform (Terraforms, 2015)
among others. From this analysis the authors
conclude that the most of the current so-called
“DevOps” solutions focus mostly on Continuous
Delivery (CD – often also named continuous
deployment in literature), while simultaneously
applying practices like Continuous Integration (CI),
Continuous Quality (CQ). CD practices allow for
automated deployment, while CQ and CI practices
allow for errors to be caught in an earlier phase of
the development cycle which are thus cheaper to
solve and with less rework, accompanied by a
configuration management system. However, the
tools hereby presented while aiming at cloud
deployments, they do not fully support the multi-
cloud approach.
1.2.2 Deployment Simulation
There are two main sources in the state of the art
related to this topic, namely the model based testing
and the cloud resource simulation.
The use of models for designing, simulating and
testing such systems is currently one of the strongest
industrial trends with significant impact on the
overall development and quality assurance
processes. Most of developments in that area, such
as Fokus!MBT (Fraunhofer Fokus, 2015) have been
mainly focused on the testing of the correctness of
the model and not so much in the determination of
the non-functional characteristics of the resulting
system. However, some works have been carried out
on the simulation of performance metrics for grid
computing (Li, 2009); with can be used as a starting
point for the definition of strategies for cloud
resources performance simulation.
In the domain of cloud resource simulation, the
most relevant input is Cloudsim (CLOUDS
Laboratory, University of Melbourne, 2012). It
provides a generalized and extensible simulation
framework that enables seamless modelling,
simulation, and experimentation of emerging Cloud
computing infrastructures and application services.
The usage of Cloudsim is based on coding the use
cases in Java, and the provided example mainly
make reference to IaaS infrastructures.
On other hand, there are several tools in the
market that allow comparing IaaS services, such as
CloudHarmony (CloudHarmony, n.d.), or
Cloudorado (Cloudorado, 2016). There are also tools
such as Cloudsleuth (Cloudsleuth, 2016) that allow
comparing the performance of IaaS and PaaS
providers, but are mostly focused in monitoring
response-time and availability. However, all these
decision support type tools usually ignore the
complexity of multi-cloud environments, where
combinations of cloud services need to be evaluated
as well as non-functional properties.
1.2.3 Modeling of Dynamic and
Reconfigurable Multi-Cloud
Applications
Multi-Cloud is often defined as the serial or
simultaneous use of services from diverse providers
to execute an application (Petcu, 2013). At business
level, Hybrid Cloud is the term commonly used.
Gartner (Mazzuca, 2015) defines hybrid Cloud as
the coordinated use of cloud services across isolation
and provider boundaries among public, private and
community service providers, or between internal
and external cloud services. A number of scenarios
demonstrate these serial or simultaneous interactions
among hybrid heterogeneous private and public
clouds and across all cloud layers (IaaS/PaaS/SaaS)
(ETSI, 2013). Multi-cloud applications engineering
as they are understood by the authors of this paper,
with the application components distributed across