Quality Aspects of Serverless Architecture: An Exploratory Study on

Maintainability

Louis Racicot

, Nicolas Cloutier

, Julien Abt

and Fabio Petrillo

Ecole Polytechnique de Montr

eal, Montr

eal, Canada

Universit

e du Qu

ebec

a Chicoutimi, Chicoutimi, Canada

Keywords:

Function as a Service, Serverless, Software Architecture, Maintainability, Cloud Computing.

Abstract:

Serverless architecture is emerging more and more popular as the tools are becoming cheap and more acces-

sible. This way of designing an architecture presents many advantages especially for computing intensive and

event-driven applications. Stateless functions are the foundation for these types of architectures, and it might

cause an impact on the maintainability of the software. In this paper, we statically analyzed 25 open-source

projects using serverless architecture to bring out metrics that applies to the different characteristics of software

maintainability. We found out that some characteristics are positively impacted whilst some other seems to

be negatively impacted. This paper thus provides ﬁndings on the current state of the projects’ maintainability

using serverless architecture.

1 INTRODUCTION

Serverless architecture is an execution model where

the provider dynamically manage the resources re-

quested by an application. Despite the name, server-

less computing still require servers. In fact, serverless

computing can be seen as leasing a server for a very

small amount of time before releasing it for another

application to use it. Most modern application use

serverless computing for CPU heavy task and ded-

icate smaller server that they can scale horizontally

for the routing. Typically, theses smaller server are

stateless so that they can send the CPU heavy tasks

to the lambda functions asynchronously and continue

with the execution until the functions return its re-

sults. This create a non blocking pipeline that is very

efﬁcient, very scalable and easily maintainable. A big

promoter of this architecture is Netﬂix, proving that

the leap of faith is actually fealisable when done cor-

rectly.

Since its inception, consumers have discovered a

growing number of use cases for the serverless archi-

tecture. The most notable use cases are automated

backups, scheduled Cron jobs, processing uploaded

object (ie: on S3), analyzing log or simply processing

and arbitrarily payload. All theses tasks are known to

be CPU extensive, but each one of them already had

one or many solutions in place. The serverless pay-

per-use business model is interesting for all of theses

task. Considering that hosting and compute power

is one of the biggest recurring cost for startup aside

from workforce, it is interesting for them to know the

trade-off of each solution in order to make the most

proﬁtable choices. Given all of the possible services

that can achieve the same results, it can be expensive

to prototype them all to assert which service is the

most suitable for a given problem. Serverless archi-

tecture is gaining popularity due to its interesting pric-

ing model. The growth in popularity of this model

certainly raises concerns as it is used as a replace-

ment for existing service architectures that are at the

moment mature, somehow scalable and maintainable

(Shadija et al., 2017).

In the recent years Amazon along with other big

technology companies have started to offer server-

less options as part of their cloud systems solutions.

Among others, there are known solutions such as

AWS Lambda (Amazon, 2017b), Microsoft Azure

Functions (Microsoft, 2017) and Google Cloud Func-

tions (Google, 2017).

As serverless architecture is fairly new, it is yet

to be proven beneﬁcial as a solution. In this paper,

we analyzed the maintainability of different serverless

project. Our main research questions is:

RQ: Does the strain of creating serverless functions

impact the maintainability?

The purpose of this paper is to identify maintain-

ability aspects in serverless based solutions and to un-

Racicot, L., Cloutier, N., Abt, J. and Petrillo, F.

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability.

DOI: 10.5220/0007842000600070

In Proceedings of the 14th International Conference on Software Technologies (ICSOFT 2019), pages 60-70

ISBN: 978-989-758-379-7

cover those ﬁndings for potential future researches.

Furthermore, we will discuss the current popular

methods for deploying serverless solutions and cur-

rent trends in conﬁguration management for this type

of project. To present our results and discussions, we

ﬁrst deﬁne what is serverless architecture. Then, we

present the related work. After that, present be pre-

sented our methodology along with our results. We

then discuss our ﬁndings. Finally, we present our con-

clusion and future work.

2 BACKGROUND

2.1 Serverless and FaaS

Serverless architecture is a model of cloud com-

puting services that cloud providers offer and man-

age dynamically allocated computational resources,

pricing by consumed resources (for example number

of requests) rather than on prepaid units of capac-

ity. Sbarski (Sbarski, 2016) deﬁned ﬁve principles of

serverless architectures:

1. Use a compute service to execute code on demand

(no servers).

2. Write single-purpose stateless functions.

3. Design push-based, event-driven pipelines.

4. Create thicker, more powerful front ends.

5. Embrace third-party services.

Function as a service (FaaS) is a type of serverless ar-

chitecture that implies a service written with a set of

individual functions which are all entry points in the

program. These functions are individually deployed

and versioned, and are stateless in the sense that each

public function call is run in a new runtime environ-

ment. Persistence is only available through external

storage solutions such as a database. In fact, Faas is

one way to implement serverless architecture.

In general, theses functions are connected to the

client services with an independent interface like a

REST API. Stateless functions are usually better for

short-run time execution even if it is possible to let

it run longer. It is usually designed to respond to a

query, process some data and give a response. It is

not designed be pending and waiting for other queries.

The ﬁgure 1 is an example of a typical function as a

service.

Compared to other solutions such as PaaS, SaaS

or IaaS (Infrastructure as a Service), FaaS is the solu-

tion that allows the most control over the code while

allowing no control at all over the infrastructure. That

way, developers are able to focus only on the code

Figure 1: Example of a Stateless Function Generic Archi-

tecture on AWS Lambda (Golden, 2016).

and its deployment when the FaaS provider takes care

of the whole server infrastructure and conﬁguration

(Baldini et al., 2017).

Notably, a serverless architecture is more cost-

efﬁcient than owning or renting a cluster due to the

periods of non-utilization that are not billed to the

consumer. A serverless architecture also make the

code development easier, since the multi-threading

and the scaling is taken care of by the provider.

In counterparts, a serverless architecture suffer from

some latency when a function has to be initiated after

not being used for a long period of time. Also, for

very high CPU intensive task, a serverless architec-

ture might not be suited due to the limited resources

allocated for each function. Finally, the consumer

does not have the control of their server. This can

make the deployment, the monitoring harder. This

also constraint the user to a small subset of languages.

2.2 Maintainability

Software maintainability can be described by “the

ease with which a software system or component can

be modiﬁed to correct faults, improve performance or

other attributes, or adapt to a changed environment”

(Committee et al., 1990).

Furthermore, the international standard ISO/IEC

25010:2011 deﬁnes maintainability as the “degree

of effectiveness and efﬁciency with which a product

or system can be modiﬁed by the intended main-

tainers ” and breaks it down into ﬁve subcategories:

modularity, reusability, analysability, modiﬁability

and testability (ISO/IEC, 2010). It also states that

maintainability includes the installation of updates

and upgrades, which, in the context of FaaS, can be

interpreted as the deployment.

Multiple deﬁnitions of maintainability by differ-

ent software quality models exist. The McCall model

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability

(Cavano and McCall, 1978) describes maintainability

using four criteria:

• Simplicity

• Conciseness

• Self-descriptiveness

• Modularity

Similarly, the Boehm’s quality model (Boehm

et al., 1976) deﬁnes maintainability as a product of

multiple characteristics

• Testability

• Understandability

• Modiﬁability

Which themselves depends upon multiple

sub-characteristics which are Consistency, Ac-

cessibility, Communicativeness, Structuredness,

Self-descriptiveness, Conciseness, Legibility and

Augmentability.

Finally, ISO 25010 (ISO/IEC, 2010) de-

scribes maintainability as a product of a few

sub-characteristics:

• Analysability

• Modiﬁability

• Testability

• Modularity

• Reusability

To help up having a better understanding of what

maintainability is, we sum up all the characteristics of

different quality models in Table 1.

Table 1: Relation between maintainability characteristics

and quality model.

Quality model

McCall

Boehm

ISO 25010

Simplicity X

Conciseness X X

Self-descriptiveness X X

Modularity X X

Testability X X

Understandability X

Modiﬁability X X

Analysability X

Reusability X

Visser (Visser et al., 2016) used ISO 25010 as

quality model to deﬁne his metrics. An interesting

part of his work is that even though simplicity and

conciseness are not even part of ISO 25010, two met-

rics (unit complexity and unit size) can account for

these characteristics. The understandability is the

only characteristic not used in Visser’s model. It

can be calculated using the three metrics deﬁned in

Nazir’s paper (Nazir et al., 2010). These metrics are

namely Inheritance, Coupling and Cohesion. How-

ever, most of the projects we analyzed don’t use

classes and therefore the inheritance metric does

not apply. Furthermore, we show in our results that

projects using serverless technologies generally have

a low coupling and are very small, which is a good

sign for a strong cohesion. Because of that, we de-

cided not to analyze understandability in detail, al-

lowing us to use only Visser’s model.

3 MAINTAINABILITY AND FaaS

In this section, we examine how a serverless archi-

tecture can impact the aspects of maintainability. For

each of them, we formulate a hypothesis. This hy-

pothesis will later on be veriﬁed based on the relation

between the characteristics of maintainability and the

empirically measurable properties of the system, as

deﬁned by Visser, as presented in Table 2.

Table 2: Relation of sub characteristics and system proper-

ties.

Volume

Duplication

Unit complexity

Unit size

Unit interfacing

Module coupling

Component balance

Component independence

Analysability X X X X

Modiﬁability X X X

Testability X X X

Modularity X X X

Reusability X X

Relation of sub-characteristics and system properties

(Visser et al., 2016)

3.1 Modularity

ISO25010 describes modularity as the “degree to

which a system or computer program is composed

of discrete components such that a change to one

ICSOFT 2019 - 14th International Conference on Software Technologies

component has minimal impact on other compo-

nents.”(ISO/IEC, 2010) If we assume that most

serverless projects are designed using this guideline,

it should imply that serverless architecture complies

with the modularity aspect of maintainability. Conse-

quently, we can formulate our ﬁrst hypothesis:

H1: Software that uses serverless technologies have a

low module coupling, their components are well bal-

anced and are independent. Thus, it complies with the

modularity aspect of maintainability.

3.2 Reusability

The reusability is the possibility of an asset to be used

by more than one system. Since serverless functions

are available by an HTTP request and that it runs

completely independently from the software that

uses it, we can suppose that serverless architecture

enforce the reusability aspect of maintainability.

As GC. Fox at al. stated, serverless is good for

short-running, stateless and event-driven codes such

as micro-services (Fox et al., 2017). Furthermore,

because of the nature of a serverless function, it only

has one interface available but does not enforce a

maximum number of parameters. However, we still

suppose that most developers keep a small number of

parameters.

H2: Software that uses serverless technologies have

small units of code with a small interface.

3.3 Analysability

The analysability of a project describes how easy it is

to diagnose and test the components of the project.

Visser states that the metrics that have an impact on

the analysability are the size of the code base, the

amount of duplicated code, the size of the units of

code and the balance of the components. Serverless

providers, though they limit the execution time of a

function, do not have any restrictions for its volume.

It is possible to run large pieces of code as FaaS,

but we suppose that most serverless code is quite

small. As said previously, we also suppose that the

unit size should be small. However, we think that

a serverless architecture does not have a positive

impact on the code duplication in the case where a

project uses multiple serverless functions. Because

all functions have to be independent, we expect to

ﬁnd the duplication of libraries and helper functions.

Finally, we suppose that serverless architecture does

not have an impact on the component balance.

H3: Software that uses serverless architecture has a

small serverless code base and small units of code,

but it has duplicate code.

3.4 Modiﬁability

The modiﬁability of a system represents how easy

it is to modify the code without introducing defects

or degrading the quality of the existing product

(Committee et al., 1990). The modularity and the

analysability inﬂuence the modiﬁability because it

includes the coding activities as well as the design,

the documentation and the validation of the new

code. Factors that inﬂuence the modiﬁability are the

amount of duplicated code, the complexity of the

code units and the coupling between the modules.

H4: Software that uses serverless technologies tends

to be easy to modify because it has simple units of

code and low coupling.

3.5 Testability

The testability is the ease to deﬁne and execute tests.

We already emit hypotheses concerning the volume

of the code, the complexity of the code units and the

component balance. However, we would like to add

some qualitative details about this aspect. While tests

can be easily deﬁned and executed locally, the local

environment might not behave exactly like the service

provider. It is especially true if we want to test the

execution time or if we have a lot of environment

speciﬁc conﬁguration. We can overcome these

problems if the deployment on a test environment

is easy to do. We will talk about this in the next

subsection.

H5: Software that uses serverless technologies are

easy to test because they have a small code base, they

have simple units of code and they are made of inde-

pendent components.

4 ASPECTS OF CODE

MANAGEMENT

4.1 Deployment

The ability to easily deploy or install an update is a

crucial part of the maintainability. We consider that

even if a code is very easy to modify, developers or

release engineers need to be able to deploy their mod-

iﬁcations quickly, without breaking the application or

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability

introducing deployment-related issues. When we are

talking about deploying a serverless function, things

can become complicated. Multiple functions can re-

quire to be deployed at the same time if their interface

or behaviour changes.

How do you deploy the application in those

cases? Do you have to deploy the function and the

client at the exact same time? What if the function

has multiple clients? Does your provider allow to

have multiple versions of the function at the same

time? And if so, would your application be ﬂexible

enough to allow the coexistence of multiple version

of the function without producing inconsistencies

in your data? These questions cannot be answered

by empirical data from the code so instead, we will

bring what we have found in the literature to verify

our hypothesis.

First of all, there are some tools that can ease

the deployment. For example, the Serverless Frame-

work(Framework, 2017) offers a range of functionali-

ties to quickly put code into production with a variety

of cloud services providers.

Another important part of the deployment is the

management of deployed versions. If multiple clients

are using FaaS, you might not want to upgrade them

all at the same time if you upload a new version of

your function that breaks the backward compatibility.

Some providers such as AWS have a very straight for-

ward versioning system that allow multiple versions

of the same function to coexist. Other providers like

Google Cloud and Microsoft Azure don’t have such

an easy way of managing versions.

In any cases, there seems to have a certain

overhead in the management of versions and thus, we

formulate the following hypothesis:

H6: The complexity of deployment of serverless soft-

ware is harmful for its maintainability.

4.2 Conﬁguration Management

Because the serverless functions are completely inde-

pendent from each other and from the client’s soft-

ware, they can be seen as a different program. They

can also be developed and maintained by different

teams, which makes each serverless function a differ-

ent project. There is also an explicit dependency be-

tween the functions and their clients, and there can be

dependencies between the serverless functions them-

selves. On a large scale, this can lead to complex de-

pendency problems likes the ones that package man-

agers have to handle, and we haven’t found a real so-

lution to that problem yet. The technologies are still

too young to be confronted to this problem, but it

is something that could happen in the future. On a

regular scale projects, however, this leads to a wide

and still unresolved question that is way beyond the

scope of this paper: mono-repository versus multi-

repository (Potvin and Levenberg, 2016; Potencier,

2016; Cavale, 2016).

Having only one repository per lambda can help

to set up continuous integration tools (Amazon,

2017a). On the other hand, it is much easier to man-

age each functions of a project in a single repository,

especially if all functions are only bound to a single

project.

H7: Conﬁguration management for software that uses

serverless technologies is still an open question.

5 STUDY DESIGN

The maintainability of a project has multiple aspects.

The most prominent one is the maintainability of the

code which can be measured (Visser et al., 2016). We

mainly investigated maintainability aspects using the

following strategy:

1. Qualitatively discuss possible maintainability and

management issues, formulating seven hypothesis

(sections 3 and 4)

2. Extract meaningful metrics from FaaS projects to

provide empirical results

3. Compare our discussion with empirical results to

evaluate the state of maintainability

4. Discuss how code source is managed within con-

ﬁguration management system and deployment

tools

To empirically analyze the impacts of FaaS on

the code maintainability, we analyzed open-source

projects from Github (Just Serverless, 2017) using

serverless architecture.

5.1 Practices and Tools

To extract the metrics, we used BetterCodeHub, an

online static analysis service whose criteria ares base

on guidelines for more maintainable code (Visser

et al., 2016). BetterCodeHub test various maintain-

ability aspects of project’s source code. The results

are organized in a way that the issues are shown to

the user of the service with speciﬁc reasons why it

failed. It has the limitations of analyzing only GitHub

ICSOFT 2019 - 14th International Conference on Software Technologies

projects and the size of the code base is limited to 100

KLOC in the free version. However, it was not a true

limitation because of FaaS projects are usually small

in terms of lines of code.

BetterCodeHub is used to test our hypothesis be-

cause it adopts the same guidelines criteria as SIG

for evaluating the maintainability (Software Improve-

ment Group, 2017; Visser et al., 2016). We can link

our hypothesis to the guidelines using Table 2.

5.2 Analyzed Serverless Projects

To test these hypotheses, we manually inspected a cu-

rated list of serverless projects and we ﬁltered them

based on the following criteria: (1) the project must

be based on the function-as-a-Service pattern; (2) the

project must be of a non-trivial in complexity and in

size; (3) the project must be analyzable by Better-

CodeHub.

With theses criteria, we found 25 projects (table 3)

varying from 1 function to 95 functions. We note that

the project with only one function has been accepted

due to its complexity. It is an SSH certiﬁcate author-

ity with test coverage (Netﬂix, 2017). The projects

are chosen to be as diverse as possible. We need to

use projects from different authors and with different

scope in order to ensure that the results won’t be bi-

ased.

To ﬁnd the number of functions, a manual inspec-

tion must be done. Depending on the framework and

its version, the information about them is not at the

same place. For example, the serverless framework is

using a yml conﬁguration ﬁle after the version 1.0 to

manage every function (Serverless Inc., 2017).

6 RESULTS

All these results have been compiled in Table 3. The

biggest offenders for the guidelines in the selected

projects are the size and simplicity of units, the du-

plication of codes and the size of the parameters. The

modules and components guidelines are nearly com-

plying.

Finally, the sum of all metrics can be reported to

the initial characteristics of maintainability as shown

on Table 4 and discussed in the next section of this

paper.

Our results show that most of analyzed projects

separate concerns in modules, Couple Architecture

Components Loosely, keep architecture components

balanced, keep the codebase small, and Write Clean

Code. However (1) 88% (22/25) of analyzed projects

do not write short units of code; (2) 60% of projects

(15/25) write complex units of code and 52%

(13/25)do not use well automated tests; ﬁnaly (3) 60%

(15/25) tend to have complex unit interfaces.

7 DISCUSSION

With theses results in mind, we discuss the aspects of

maintainability and management, evaluating our hy-

potheses.

7.1 Modularity

H1: Software that use serverless technologies have a

low module coupling, their components are well bal-

anced and are independent. Thus, it complies with the

modularity aspect of maintainability.

Serverless Programming is at its core a way to

make services. We expect theses small services to

increase modularity compared to a monolithic multi-

layer application (Shadija et al., 2017). They ensure

a strong separation of the different behaviours with

an interface and that they are all self-contained. Thus

we are not surprised that our results seem to conﬁrm

a good modularity. All the different criteria for the

ﬁrst hypothesis are validated on the majority of the

projects analyzed.









Software that use serverless technologies have a

low module coupling, their components are well

balanced and are independent.

7.2 Reusability

H2: Software that use serverless technologies have

small units of code with a small interface.

It seems that serverless projects tend to have big

units of code and the majority of the analyzed projects

do not respect the required criteria. However, we note

that a bigger project, such as MoonMail, does not

mean a greater chance of facing this issue. Server-

less architecture should not prevent code reusability.

It’s still possible to split the code in multiple small

units of code that are reusable. But it doesn’t enforce

or encourage that so it needs to come from within.









Serverless projects tend to have big units of code.

7.3 Analysability

H3: Software that use serverless architecture has a

small serverless code base and small units of code but

do have duplicate code.

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability

Table 3: BetterCodeHub guidelines applied on serverless projects.

Write Short Units of Code

Write Simple Units of Code

Write Code Once

Keep Unit Interfaces Small

Separate Concerns in Modules

Couple Architecture Components Loosely

Keep Architecture Components Balanced

Keep Your Codebase Small

Automate Tests

Write Clean Code

microapps/MoonMail 3 3 7 3 3 3 3 3 3 3

jonatasschagas/langadventurebackend 7 7 3 3 3 3 7 3 7 3

craftship/codebox-npm 7 3 7 7 3 3 3 3 3 3

C0k3/session 7 7 7 7 3 7 3 3 3 7

agentmilindu/Serverless-Pre-Register 7 7 7 3 3 3 3 3 7 3

haw-itn/serverless-web-monitor 7 3 7 7 3 3 3 3 7 3

bart-blommaerts/serverless garage 3 3 7 7 3 3 3 3 7 3

michalsanger/serverless-facebook-messenger-bot 7 3 7 7 3 3 3 3 7 3

laardee/serverless-authentication-boilerplate 7 3 3 3 3 3 3 3 3 3

airbnb/binaryalert 3 3 3 3 3 3 3 3 3 3

airbnb/streamalert 7 7 3 7 3 3 3 3 3 3

Netﬂix/bless 7 7 3 7 3 3 3 3 3 3

apache/incubator-openwhisk 7 7 3 7 3 7 3 3 7 3

fnproject/fn 7 7 3 7 3 3 7 3 7 3

capitalone/cloud-custodian 7 7 3 7 3 7 3 3 3 3

blockstack/blockstack-core 7 7 7 7 3 3 3 3 3 3

adieuadieu/serverless-chrome 7 7 3 7 3 3 3 3 7 7

danilop/LambdAuth 7 7 7 7 3 3 3 3 7 3

serverless-heaven/serverless-webpack 7 7 7 3 3 3 3 3 3 3

bcongdon/corral 7 3 3 7 3 3 3 3 3 3

awslabs/aws-serverless-auth-reference-app 7 3 7 7 3 3 3 3 7 3

open-lambda/open-lambda 7 7 3 7 3 3 7 3 7 7

0x4D31/honeyLambda 7 7 3 3 3 3 7 3 7 3

amplify-education/serverless-domain-manager 7 7 7 3 3 3 3 3 3 3

awslabs/serverless-photo-recognition 7 3 3 3 3 3 7 3 7 3

Successes 3 10 13 9 25 22 20 25 12 22

Failures 22 15 12 16 0 3 5 0 13 3

Table 4: Validation of maintainability characteristics.

Analysability X

Modiﬁability X

Testability X

Modularity X

Reusability 7

Deployment X

Conﬁguration management 7

As expected, the duplication of code is an issue

encountered in our analysis. Half the projects encoun-

tered this problem. As described in H2, the units of

code are bigger than expected. Analysability may be

an issue for serverless applications.









Analysability may be an issue for serverless ap-

plications.

7.4 Modiﬁability

H4: Software that use serverless technologies tends

to be easy to modify because it has simple units of

ICSOFT 2019 - 14th International Conference on Software Technologies

code and low coupling. However, there is a lot of

duplicated code.

It does not seem like serverless technologies tend

to be easy to modify. The metric about simple units

of code and code only being written once looses by a

week margin. For example, the small authentication

service done by Coca-Cola has big units of code con-

taining all the logic of a public function. Since the

projects and the use cases are fairly small, it may be

tempting for developers to put everything in the same

unit and not to split the code base correctly. It may

become an issue with small projects which are scal-

ing more than the initial scope.









It is not clear whether serverless technologies

tend to be easy to modify.

7.5 Testability

H5: Software that use serverless technologies are

easy to test because they have a small code base, they

have simple units of code and they are made of inde-

pendent components.

The code base is small and the components are

independent. Still, we see a good advantage for testa-

bility in a FaaS application. Another aspect that is

not covered by the results is the nature of a serverless

application. Since every public function is contained

and is run in a new runtime, the tests of a function are

simpliﬁed because the combinations of possible states

for a function are reduced. The dependencies are only

coming from the input parameters and from external

sources, like a database or from networking.









Serverless technologies are easy to test because

they have a small code base and they are made of

independent components. Although tests are not

always automated

7.6 Deployment

H6: The complexity of deployment of serverless soft-

ware is harmful for its maintainability.

While we have our worries about deployments,

we are seeing solutions implemented by the producers

and the frameworks to mitigate theses issues. For ex-

ample, the serverless framework is offering important

features to ease the deployment. They implemented

a command line which deploy your functions on your

environment. This ease and accelerate the process,

which is important, but it permits automation and the

use of conﬁguration ﬁles reduce the possibilities of

human errors during a deployment.

Listing 1: Example of a conﬁguration ﬁle used in the

Serverless Framework

s e r v i c e : s e r v i c e −name

p r o v i d e r :

name : aws

s t a g e : b e t a

r e g i o n : us−w est −2

Another important aspect is the possibility to

choose a stage target, the targets can be things like

beta, prod, v1, v1 5, v2, etc. This other feature gives

the possibility to manager multiple versions of the

same API and ensure compatibility between services

which are not updated at the same time. (Serverless

Inc., 2017).









The deployment of serverless software is in gen-

eral simpliﬁed by offering features to ease the de-

ployment.

7.7 Conﬁguration Management

H7: Conﬁguration management for software that uses

serverless technologies is still an open question.

As mentioned in the previous section, there is

still a large debate on many-repositories versus mono-

repository. The projects we analyzed always had all

their functions in the same repository. This is in fact

much easier to manage by developers and maintain-

ers. Google argues about the reasons why they are us-

ing a single repository for most of their code and the

speciﬁc branching strategy and development work-

ﬂow they are using in order to make it work (Potvin

and Levenberg, 2016). Other companies like amazon

suggest breaking the project repository into smaller

ones for large projects. (Workﬂow, 2017)

All of this seems to point to the facts that this is

still an open question. Conﬁguration management has

a great impact on the deployment and development

process and thus, on many aspects of software quality

including maintainability.









Serverless conﬁguration management is an open

question, and there is still a large debate on

many-repositories versus mono-repository.

7.8 Research Opportunities

Serverless is a recent topic, and there are several re-

search opportunities to explore.

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability

Performance. This paper was mostly about only

one aspect of software quality. Other quality factors

such as performance of FaaS in general and between

different providers could be a good tool to help the

decision-making process of what component to put in

a serverless function and with which provider. To go

even deeper in performance, a cost beneﬁces analysis

(CBA) framework, based on the resources that can be

saved, could be made for serverless function to help

CTOs decide when to use this kind of technology.

Stability. Another important metric beyond the

scope of this paper that could inﬂuence maintainabil-

ity is the stability of the serverless functions over

time. Raemaekers, Deursen and Visser (Raemaek-

ers et al., 2012) have developed an interesting frame-

work to analyze the stability of software over time that

could probably be adapted to serverless functions.

Conﬁguration Management. As found previously

in this paper, conﬁguration management for server-

less technologies is still an open question. It would

be interesting to point out the beneﬁts and disadvan-

tages of each conﬁguration management solution for

serverless technologies to guide technical managers

and developers in the design of their solution.

Continuous Integration. Continuous integration

has been made easy for monolithic projects with tools

such as Jenkins. When it comes to serverless projects,

in which multiple functions sometimes written in dif-

ferent languages can be used, it is not that clear on

how to conﬁgure and perform continuous integration.

This could be another interesting research question.

8 THREATS OF VALIDITY

One of the biggest issues with our analysis is the lim-

ited number of public projects using a serverless ar-

chitecture. A lot of the projects we found are sim-

ple demos as trivial as an application printing “Hello

World,” which are unusable for research like ours.

The projects that we analyzed for the most part con-

tains fewer than 10 KLOC of code and may not be

a good representation of maintainability on bigger

projects. MoonMail was the biggest one we found,

but probably companies are making bigger applica-

tions which are not open-source. However, this limi-

tation is because an immaturity of serverless domain,

and we are investigate the current state of practice in

serverless domain. Thus, I plan redo this study in the

future to include more mature projects and also in not

only open-source projects.

Another issue in our paper is the absence of em-

pirical data on deployment. In this paper we analyzed

about deployment in a qualitative way, however, no

real data have been found on the subject.

Finally, the quality of your depends of Better-

CodeHub outputs. That threats is mitigated because

the tool is developed by experts in software quality,

but more deep studies should be performed to conﬁrm

that claim.

9 RELATED WORK

Serverless programming is quite new in the program-

ming model world and there is not yet much academic

literature about it and a lot of open questions on the

subject.

Fox et al. (Fox et al., 2017) described server-

less technologies and their evaluation as immature

by pointing out the fact that serverless code is com-

plicated to debug and the serverless debugging tech-

nologies are non-existent. They also stated that most

serverless programs are not easily portable to another

FaaS provider. Despite these drawbacks, serverless

patterns tend to be much more scalable and cost-

effective.

A more recent paper (Baldini et al., 2017) brings

up about the current trends and opened problems of

serverless computing. At the moment, serverless plat-

forms tend to limit developers to their speciﬁc ecosys-

tem. This may change in the future as open source

solutions such as OpenLambda(OpenLambda, 2016)

appears and offer a framework which can deploy the

solution of multiples cloud services. According to the

article, the best use cases of FaaS are CPU intensive

and event-driven applications. The best programming

model for FaaS would be very similar to functional

reactive programming, with small, stateless and idem-

potent functions. Finally, that paper pointed out some

of interesting challenges that may inﬂuence the main-

tainability of serverless based solutions. These chal-

lenges include deployment, composability and code

granularity.

Being relatively recent, serverless computing is

just starting to see get some attention by the research

community. A lot of the research has been trying

to deﬁne what is serverless computing and Function-

as-as-Service (FaaS) (Jonas et al., 2017)(Spillner,

2017)(Varghese and Buyya, 2017). Some work has

been done to benchmark the performance of server-

less architectures. It has been shown by (Jonas et al.,

2017) that it is possible to build a model that is general

ICSOFT 2019 - 14th International Conference on Software Technologies

enough to implement a number of distributed com-

puting model such as Bulk synchronous parallel. In

fact, they show that highly parallelizable operation,

such as matrix multiplication, can be parallelized us-

ing lambda with very low bottleneck. More precisely,

they built their own framework to serialize any highly

parallelizable task and send them to an arbitrary num-

ber of workers on lambda. They show that the ag-

gregates TFLOPS scale linearly with the number of

workers. They also show that the read/write through-

put scale also linearly with the number of workers. Fi-

nally, they break down the time taken for each phase

of the lambda function. They show that the initial-

ization bottleneck of the lambda is 15%. In certain

application this bottleneck can be considerable.

Malawski et al. (Malawski et al., 2018) they com-

pare the serverless architecture with the HyperFlow

architecture and evaluate their benchmark on AWS,

Google Cloud and IBM OpenWhisk. They execute

their benchmark on Mersenne Twister and Linpack.

They compare the CPU resources allocated with re-

spect to the memory allocated. They observe that the

CPU resources allocated on AWS scale linearly with

the memory allocated, where the CPU resources al-

located does not scale linearly with the memory allo-

cated on Google cloud framework. They also observe

that AWS achieve over 30 GFLOPS where Google

cloud tops at 17 GFLOPS. This difference is due to

the hardware used on each platform.

10 CONCLUSION

The goal of this paper was to determine whether or

not the efforts of using serverless functions is worth

in terms of software maintainability. We think that

we have achieved this goal by providing an empiri-

cal analysis on metrics we could calculate on static

code. While taking account of the threats to the valid-

ity of this study, we still see a tendency for an increase

of maintainability in Function-as-a-Service architec-

tures. We also wanted to promote more research on

the subject and thus gave many ideas on possible fu-

ture researches.

Our results show that more than 100% (25/25) of

analyzed projects separate concerns in modules, Cou-

ple Architecture Components Loosely, keep architec-

ture components balanced, keep the codebase small,

and Write Clean Code. However (1) 88% (3/25) of

analyzed projects do not write short units of code; (2)

60% of projects (15/25) write complex units of code

and do not use well automated tests.

In fact, we found that (1) software that use server-

less technologies have a low module coupling, their

components are well balanced and are independent;

(2) serverless projects tend to have big units of code;

(3) analysability may be an issue for serverless appli-

cations; (4) it is not clear serverless technologies that

tend to be easy to modify; (5) serverless technolo-

gies are easy to test because they have a small code

base and they are made of independent components;

(6) the deployment of serverless software is in gen-

eral simpliﬁed by offering features to ease the deploy-

ment; and (7) serverless conﬁguration management is

an open question, and there is still a large debate on

many-repositories versus mono-repository.

We also mentioned about deployment and conﬁg-

uration management, which are in many ways related

to maintainability. We discussed on how deployment

can be a challenge and what seems to be the possible

solutions. We also found out that conﬁguration man-

agement is still a debate.

Our major concerns for maintainability are on

reusability and on conﬁguration management. In the

project we analyzed, there seems to have a lot of du-

plicated code and heavy units of code with big inter-

faces. It would be interesting to know if these bad

tendencies are also seen in commercial projects that

might make a heavier use of FaaS technologies.

We hope that more research will be done in the

future on the subject to analyze subsequent, large-

scale applications that exist in a serverless way and

in a regular way to compare our results. This way, we

could really compare what the advantages and disad-

vantages brought by serverless architecture are in a

more meaningful and precise way.

REFERENCES

Amazon (2017a). Continuous integration deployment for

aws lambda functions with jenkins and grunt. https:

//aws.amazon.com/blogs. (Accessed on 10/17/2017).

Amazon (2017b). Serverless computing - amazon web ser-

vices. https://aws.amazon.com/serverless/. (Accessed

on 09/20/2017).

Baldini, I., Castro, P. C., Chang, K., Cheng, P., Fink, S. J.,

Ishakian, V., Mitchell, N., Muthusamy, V., Rabbah,

R. M., Slominski, A., and Suter, P. (2017). Serverless

computing: Current trends and open problems. CoRR,

abs/1706.03178.

Boehm, B. W., Brown, J. R., and Lipow, M. (1976). Quan-

titative evaluation of software quality. In Proceedings

of the 2Nd International Conference on Software En-

gineering, ICSE ’76, pages 592–605, Los Alamitos,

CA, USA. IEEE Computer Society Press.

Cavale, A. (2016). Our journey to mi-

croservices and a mono repository.

https://www.shippable.com/index.html.

Quality Aspects of Serverless Architecture: An Exploratory Study on Maintainability

Cavano, J. P. and McCall, J. A. (1978). A framework for

the measurement of software quality. SIGSOFT Softw.

Eng. Notes, 3(5):133–139.

Committee, I. S. C. et al. (1990). Ieee standard glossary

of software engineering terminology (ieee std 610.12-

1990). los alamitos. CA: IEEE Computer Society, 169.

Fox, G. C., Ishakian, V., Muthusamy, V., and Slominski, A.

(2017). Status of serverless computing and function-

as-a-service (faas) in industry and research. arXiv

preprint arXiv:1708.08028.

Framework, S. (2017). https://serverless.com/. (Accessed

on 11/19/2017).

Golden, B. (2016). The roadmap to serverless computing:

Are you prepared? https://techbeacon.com/roadmap-

serverless-computing-are-you-prepared. (Accessed

on 11/18/2017).

Google (2017). Cloud functions - serverless environment

to build and connect cloud services. https://cloud.

google.com/functions/. (Accessed on 11/17/2017).

ISO/IEC (2010). Iso/iec 25010 system and software quality

models. Technical report.

Jonas, E., Pu, Q., Venkataraman, S., Stoica, I., and Recht,

B. (2017). Occupy the cloud: Distributed computing

for the 99%. In Proceedings of the 2017 Symposium

on Cloud Computing, SoCC ’17, pages 445–451, New

York, NY, USA. ACM.

Just Serverless (2017). Github - justserverless/awesome-

serverless: Curated list of resources related to

serverless architectures and the serverless frame-

work. https://github.com/JustServerless/awesome-

serverless#projects--services. (Accessed on

09/23/2017).

Malawski, M., Figiela, K., Gajek, A., and Zima, A.

(2018). Benchmarking heterogeneous cloud func-

tions. In Heras, D. B. and Boug

e, L., editors, Euro-

Par 2017: Parallel Processing Workshops, pages 415–

426, Cham. Springer International Publishing.

Microsoft (2017). Azure functions—serverless archi-

tecture. https://azure.microsoft.com/en-ca/services/

functions/. (Accessed on 11/17/2017).

Nazir, M., Khan, R. A., and Mustafa, K. (2010). A met-

rics based model for understandability quantiﬁcation.

arXiv preprint arXiv:1004.4463.

Netﬂix (2017). Netﬂix/bless: Repository for bless, an ssh

certiﬁcate authority that runs as a aws lambda func-

tion. https://github.com/Netﬂix/bless. (Accessed on

10/17/2017).

OpenLambda (2016). https://open-lambda.org/. (Accessed

on 09/23/2017).

Potencier, F. (2016). A monorepo vs manyrepos.

Potvin, R. and Levenberg, J. (2016). Why google stores

billions of lines of code in a single repository. Com-

munications of the ACM, 59(7):78–87.

Raemaekers, S., van Deursen, A., and Visser, J. (2012).

Measuring software library stability through historical

version analysis. In Software Maintenance (ICSM),

2012 28th IEEE International Conference on, pages

378–387. IEEE.

Sbarski, P. (2016). Serverless Architectures on AWS. Man-

ning Publications Co.

Serverless Inc. (2017). Serverless - the serverless ap-

plication framework powered by aws lambda and

api gateway. https://serverless.com/. (Accessed on

10/17/2017).

Shadija, D., Rezai, M., and Hill, R. (2017). Towards an

understanding of microservices. In 2017 23rd Inter-

national Conference on Automation and Computing

(ICAC), pages 1–6.

Software Improvement Group (2017). Better code

hub. https://bettercodehub.com/. (Accessed on

09/23/2017).

Spillner, J. (2017). Snafu: Function-as-a-service

(faas) runtime design and implementation. CoRR,

abs/1703.07562.

Varghese, B. and Buyya, R. (2017). Next generation

cloud computing: New trends and research directions.

CoRR, abs/1707.07452.

Visser, J., Rigal, S., van der Leek, R., van Eck, P., and Wi-

jnholds, G. (2016). Building Maintainable Software,

Java Edition: Ten Guidelines for Future-Proof Code.

” O’Reilly Media, Inc.”.

Workﬂow, S. F. G. (2017). https://serverless.com/

framework/docs/providers/aws/guide/workﬂow/. (Ac-

cessed on 11/19/2017).

ICSOFT 2019 - 14th International Conference on Software Technologies