Security Testing of RESTful APIs with Test Case Mutation
S
´
ebastien Salva and Jarod Sue
LIMOS - UMR CNRS 6158, Clermont Auvergne University, UCA, France
Keywords:
RESTful APIs, Security, Test Case Generation, Test Case Mutation.
Abstract:
The focus of this paper is on automating the security testing of RESTful APIs. The testing stage of this specific
kind of components is often performed manually, and this is yet considered as a long and difficult activity.
This paper proposes an automated approach to help developers generate test cases for experimenting with
each service in isolation. This approach is based upon the notion of test case mutation, which automatically
generates new test cases from an original test case set. Test case mutation operators perform slight test case
modifications to mimic possible failures or to test the component under test with new interactions. In this paper,
we examine test case mutation operators for RESTful APIs and define 18 operators specialised in security
testing. Then, we present our test case mutation algorithm. We evaluate its effectiveness and performance on
four web service compositions.
1 INTRODUCTION
One of the key motivations for software security is
the prevention of attackers exploiting software flaws,
which can lead to compromising application security
or revealing user data. Despite the continuous growth
of the security testing market, there is still an inad-
equate emphasis on this activity, exposing organisa-
tions and end users to unforeseen risks when using
vulnerable systems or software. One aspect that may
account for this observation, is that selecting security
solutions and crafting specific security test cases are
two tasks of the software life cycle that demand time,
expertise, and experience. Developers often lack the
guidance, resources, or skills on how to design, im-
plement secure applications, and test them.
A way to help developers in security testing is the
use of test automation, which addresses challenges re-
lated to time constraints, complexity, and coverage.
In this scope, model based testing (Li et al., 2018) of-
fers the advantage of automating the test case genera-
tion. But models are often manually written, and this
task is considered as long, difficult and error-prone,
even for experts. Numerous others approaches have
been proposed to generate test cases without speci-
fication, for example by using random testing (Ar-
curi et al., 2011), graphical user interfaces exploration
(Salva and Zafimiharisoa, 2014; Ferreira and Paiva,
2019) or automated penetration testing (Abu-Dabaseh
and Alshammari, 2018). Despite their significant ben-
efits, a recurring limitation observed in employing
these approaches is the insufficient understanding of
the application business logic and context. As a re-
sult, they may fail to identify certain security vulnera-
bilities that require a deeper understanding of how the
application behaves.
Focusing on this background, we propose an in-
termediate solution based upon the notion of test case
mutation. Unlike mutation testing that aims at evalu-
ating the effectiveness of an existing test case set by
introducing intentional errors into the original source
code of an application under test (Papadakis et al.,
2019), test case mutation automatically generates new
test cases from an original test case set. As the
original test cases should encode some knowledge
about the application under test, the mutated test cases
should deeper cover the application behaviours and
features and hence should detect further defects. A
test case mutation operator performs slight test case
modifications to mimic possible failures or to exper-
iment the system under test with new interactions.
Some test case mutation based approaches have been
proposed for detecting bugs or crashes (Xuan et al.,
2015; Xu et al., 2010; Arcuri, 2018; Arcuri, 2019;
K
¨
oroglu and Sen, 2018; Paiva et al., 2020). None of
them deals with security testing.
In this paper, we propose a new approach, specif-
ically designed for testing the security of RESTful
APIs in isolation. This firstly implies that we pro-
pose new specific mutation operators devoted to de-
582
Salva, S. and Sue, J.
Security Testing of RESTful APIs with Test Case Mutation.
DOI: 10.5220/0012698600003687
Paper published under CC license (CC BY-NC-ND 4.0)
In Proceedings of the 19th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2024), pages 582-589
ISBN: 978-989-758-696-5; ISSN: 2184-4895
Proceedings Copyright © 2024 by SCITEPRESS – Science and Technology Publications, Lda.

tecting vulnerabilities. This also means that our ap-
proach generates new executable test cases but also
mock components. We recall that a mock component
aims at simulating an existing component, while be-
having in a predefined and controlled way to make
testing more effective and efficient. Mocks are often
used by developers to make test development easier or
to increase test coverage. They may indeed be used
to simplify the dependencies that make testing dif-
ficult (e.g., infrastructure or environment related de-
pendencies). Besides, mocks are used to increase test
efficiency by replacing slow-to-access components.
In summary, the main contributions of this paper in-
clude:
1. a study on mutation operators specialised in the
security testing of RESTful APIs, including the
definition of 18 operators,
2. an algorithm for the generation of mutated test
cases along with test scripts and mock compo-
nents,
3. the implementation of the approach, along with 4
RESTful API compositions and Log files publicly
available in (Sue and Salva, 2024).
The paper is organised as follows: We study and
propose test case mutation operators for RESTful
APIs in Section 2. Our test case mutation algorithm
is presented in Section 3. Section 4 summarises our
contributions and draws some perspectives for future
work.
2 TEST CASE MUTATION
OPERATORS FOR RESTful APIs
This paper focuses on test case mutation operators de-
signed to detect vulnerabilities in RESTful APIs. This
testing context introduces specific requirements and a
testing architecture, both of which are subsequently
presented. From this architecture, we present how
test cases are modelled with Input Ouput Transition
Systems (IOTSs) and provide an illustrative example.
Then, we study the mutation operators that can be de-
fined within this scope.
2.1 Assumptions
We consider the test architecture depicted in Figure
1, whose attributes are expressed with the following
realistic assumptions:
• Black Box Testing: we employ a black box per-
spective, enabling to interact with a RESTful API,
Figure 1: Black-box test architecture for experimenting
RESTful API in isolation.
denoted SUT , only with HTTP requests or re-
sponses. We call them (communication) events;
• Event Content: observers are able to get all the
events related to a RESTful API under test along
with their contents (no encryption). In particu-
lar, events include parameter assignments allow-
ing to identify the source and the destination of
each event. Besides, an event can be identified ei-
ther as a request or a response;
• Test in Isolation: we consider conducting tests
in an isolated environment. If the RESTful API
is dependent to other services, the later shall be
replaced by mock components. We do not assume
that those mock components exist, our approach
builds them.
2.2 IOTS Test Case Definition
Given the test architecture of Figure 1, we consider
that events have the form e(α) with e some label,
e.g., a path or a status; ”*” is a special notation rep-
resenting any label. α is an assignment of param-
eters in P to a value in the set of values V . These
parameters allow the encoding of some specific web
service characteristics e.g., if an event is a request,
the receiver and sender of this request, etc. We
write x := ∗ the assignment of the parameter x with
an arbitrary element of V , which is not of interest.
E denotes the event set. We also use these addi-
tional notations on an event e(α) to make our algo-
rithm more readable: f rom(e(α))) (reps. to(e(α)))
denotes the source (resp. the destination) of the
event. isreq(e(α)), isresp(e(α)) are boolean expres-
sions expressing the nature of the event. body(e(α)),
header(e(α)), status(e(α)) are expressions returning
values in α.
We model a test case with a deterministic IOTS
having a tree form and whose terminal states express
test verdicts, e.g., pass or inc, which stands for in-
conclusive. A test step corresponds to an IOTS tran-
sition q
e(α),l
−−−→ q
′
with e(α) some event and l a label
set, which may be empty. Furthermore, we use the
notation θ labelled on transitions to represent the ab-
sence of reaction from a service under test (Phillips,
1987). Classically, we call a sequence of test steps
Security Testing of RESTful APIs with Test Case Mutation
583

a test sequence. The label set allows to easily ex-
press some knowledge about the event. For instance,
”crash” is used when the HTTP status 500 is received.
The special label ”mock” identifies events performed
by some other dependee services. Since we assume
testing SUT in isolation, the dependee services will
have to be replaced with mock components.
An IOTS test case has to met a few restrictions
to avoid indeterministic behaviours while testing. To
this end, a test case must allow at most one input event
at any state. In reference to (Tretmans, 2008), this last
restriction, we say that a test case is input restricted.
Additionally, still in the context of isolation testing
and to keep control of the testing process, a mock
component should be deterministic and return at most
one response after being invoked with the same event.
We say that a test case has to be mock response re-
stricted. This is formulated with:
Definition 1 : A test case tc is a deterministic IOTS
⟨Q,q0,Σ ∪ {θ},L,→⟩ where:
• Q is a finite set of states; q0 is the initial state;
• Σ ⊂ E is the finite set of events. Σ
I
⊆ Σ is the finite
set of input events beginning with ”?”, Σ
O
⊆ Σ is
the finite set of output events beginning with ”!”,
with Σ
O
∩ Σ
I
=
/
0;
• L is a set of labels;
• →⊆ Q × Σ ∪ {θ} × L
∗
× Q is a finite set of tran-
sitions. A transition (q, e(α),l,q
′
) is also denoted
q
e(α),l
−−−→ q
′
;
• Q
f
= {pass, f ail,inc} ⊂ Q is the set of verdict
states; if q
e(α),l
−−−→ q
f
with q
f
∈ Q
f
, then e(α) ∈
Σ
O
∪ θ;
• tc has no cycles except those in states of Q
f
;
• tc is input restricted i.e. ∀q ∈ Q : event(q) = Σ
O
∪
{e(α)} for some e(α) ∈ Σ
I
or event(q) = Σ
O
∪{θ}
with event(q) = {e(α) | ∃q
′
∈ Q : q
e(α),l
−−−→ q
′
};
• tc is mock response restricted i.e. ∀q ∈ Q :
|{q
e(α),l
−−−→ q
′
| isResp(e(α)) ∧ mock ∈ l}| ≤ 1.
Figure 2: IOTS Test Case example.
An IOTS test case example is illustrated in Fig-
ure 2. It checks whether a RESTful API AccMan can
be called with ”/checkAccountRisk”. This service is
dependent on another service called CheckRisk. The
events related to CheckRisk are labelled by ”mock” to
express that a mock component has to be built to test
AccMan in isolation.
IOTS test cases can be written manually, but this
activity may be long and error-prone, especially for
un-experimented developers. To solve this problem,
we proposed in (Salva and Sue, 2023) an approach
and tool for generating IOTS test cases from Log files.
The approach also allows to recognise some specific
behaviours (authentication, token generation, crash, )
and adds on test steps the following labels ”login”,
”token”, ”token generation”, ”crash”.
2.3 Mutation Operators for Security
Testing
The literature does not cover the use of test case muta-
tion operators specialised in assessing the security of
web services. Consequently, we initially conducted
a literature review to collect relevant data about the
security testing of RESTFul APIs. We searched for
papers indexed in online sources (Scopus, Science
Direct, IEEE Xplore, ACM Digital Library, Google
Scholar). We identified relevant papers via keyword
search by using the terms ”web services security vul-
nerabilities attacks” and then, terms ”microservice
security vulnerabilities attacks”. We found 42 and
35 works between 2006-2023. We isolated 24 pa-
pers and 3 surveys by using their abstracts and ti-
tles. We then crossed these results with the databases
CAPEC (CAPEC, 2024) and CWE (CWE, 2024) of
the MITRE organisation in order to classify attacks
and avoid duplicates. With regard to our black box
test architecture, we kept the attacks related to these
domains:
• CAPEC-21: Exploitation of Trusted Identifiers
• CAPEC-22: Exploiting Trust in Client
• CAPEC-63: Cross-Site Scripting (XSS)
• CAPEC-151: Identity Spoofing
• CAPEC-153: Input Data Manipulation
• CAPEC-115: Authentication Bypass
• CAPEC-125: Flooding
• CAPEC-278: Service Protocol Manipulation
• CAPEC-594: Traffic Injection
At this step, we collected a total number of 36
attacks. We finally augmented this compilation, by
incorporating 7 recommendations provided in the
ENISA good practice guide (Skouloudi et al., 2018).
Then, we studied these 43 elements to extract muta-
tion operators. During this process, we applied the
following criteria:
ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering
584

• C1: in accordance with our test architecture,
we build mutation operators applicable to unen-
crypted events;
• C2: a mutation operator performs small changes,
it is here used to build an attack executed with one
test case only. Hence, complex attack scenarios
cannot be considered;
• C3: knowledge typically plays a crucial role in
performing security attacks. We consider having
labels in test steps allowing to recognise authenti-
cation processes, token generation and errors. Ad-
ditional labels allow to recognise the existence of
variables acting as tokens or session identifiers;
• C4: an operator can derive new test cases and new
mock components.
Using these criteria, we finally wrote 18 mutation
operators tailored to testing in isolation the security
of black box RESTful APIs. These operators are out-
lined in Table 1, where column 2 gives short descrip-
tions, columns 3 and 4 give the expected behaviours
that should be observed after the execution of mutated
test steps and conditions on the application of the op-
erators.
3 TEST CASE MUTATION
Figure 3: Approach Overview.
As illustrated in Figure 3, we propose a test case mu-
tation approach and a tool for RESTful APIs, consist-
ing of three main stages :
1. our approach takes either existing IOTS test cases,
or Log files that are used to generate IOTS test
cases. As stated previously, the paper (Salva and
Sue, 2023) presents algorithms and a tool for per-
forming this step;
2. mutation operators are applied on IOTS test cases
to perform slight modifications that aim to mimic
security attacks. These modifications may result
in numerous mutated test cases. To address this,
we suggest strategies to restrict their generation.
Mutation operators are then applied on test cases:
we check whether the test steps meet some mu-
tation conditions to restrict the transformations on
the relevant steps only; we modify the original test
cases and complete them with tests steps and ver-
dicts to get new IOTS mutants;
3. the mutants are finally converted into test scripts
and mock components, which will be used to
check whether SUT is vulnerable.
We formalise those steps in the remainder of this
section.
3.1 Test Mutation Operator Definition
A mutation operator M of an IOTS test case tc is
made up of three elements. The first is the function
Condition, which aims at restricting the application
of the operator to some events of tc. The next func-
tion Change applies the mutation on tc and produces
an initial mutant tc
m
. Finally, Expected is a func-
tion that completes tc
m
with test sequences finished
by verdict states to express the expected observations
after the execution of a mutated event.
Definition 2 (Mutation Operator) : A Mutation op-
erator is the tuple (Condition,Change, Expected)
such that :
• Condition : Q × Σ × L
∗
× Q → {true, f alse} is a
function that expresses restrictions on test steps,
• Change : IOT S → IOT S is a mutant derivation
function,
• Expected : IOT S → IOTS is a mutant comple-
tion function, such that for any test sequence
q0
(e
1
(α
1
),l
1
)...(e
k
(α
k
),l
k
)
−−−−−−−−−−−−−→ q of the IOTS, q ∈ Q
f
is
a final state.
The function Condition(q
e(α),l
−−−→ q
′
) of a mutation
operator M may be used on the label e, on the assign-
ments α, or on the label list l. This function can be
used to define a generic operator, for example with
a condition of the form e == ∗ supplemented with
some conditions on α. But, a more specific operator
can also be defined with a condition on precise events
and parameters. The last column of Table 1 provides
several condition examples.
Change(tc) applies the mutation operator on a test
case tc and returns a mutant. We here assume hav-
ing some transitions marked with the special label
”mutation”, which targets the transitions to transform.
Given under the form of a procedure, Change could
have the following form :
Reach a transition t := q
e(α),{mutation}
−−−−−−−−→ q
1
;
Modify t;
( possibly )Keep the next outgoing transitions from q
1
to q
k
such
that to(q
k
e
k
(α
k
)
−−−−→ q
l
) = SUT ;
Prune the useless transitions from q
k
to a terminal state ;
Security Testing of RESTful APIs with Test Case Mutation
585

Expected(tc
m
) completes a mutant returned by
Change with new test steps such that the last test steps
end by a verdict state. Column 3 of Table 1 sum-
marises the test steps that are added for every muta-
tion operator.
3.2 Test Case Generation
The test architecture of Figure 1 emphasises the con-
trol and observation logics. The controller parts have
the capability to send events to SUT . These events
are those that can be modified by mutation operators
to send unexpected requests or attacks. The observer
parts will be used to collect responses, which are in-
terpreted to decide whether SU T is vulnerable or not.
In this context, we say that a test step q
e(α),l
−−−→ q
′
of a test case is mutable if the recipient of the event
is SUT itself and if the operator M may be applied
on this test step. Likewise, we use the notation
mutable(M) in tc to get the set of test steps on which
the mutation operator M can be applied. It is worth
noting that this set may be empty. This is captured by
the following definition:
Definition 3 (Mutable Test Step) : Let M be a mu-
tation operator, tc be an IOTS test case for the service
SUT , and q
e(α),l
−−−→ q
′
∈→ be a test step.
• q
e(α),l
−−−→ q
′
is mutable
M
iff to(e(α)) =
SUT ∧ M.Condition(q
e(α),l
−−−→ q
′
) ∧ ((e(α) ∈
σ
I
∨ ”mock” ∈ l)).
• mutable(M) in tc =
de f
{q
e(α),l
−−−→ q
′
∈ tc | q
e(α),l
−−−→
q
′
is mutable
M
}
Furthermore, we define the IOTS operator mark,
which simply adds a label ”mutation” on the mutable
test steps.
Definition 4 (IOTS Operator mark) : Let t =
q
e(α),l
−−−→ q
′
be a test step of a test case tc = ⟨Q,q0,Σ ∪
{θ},L,→⟩.
mark t in tc = ⟨Q
2
,q0,Σ
2
∪ {θ},L
2
,→
2
⟩ is the
IOTS test case derived from the test case tc where
Q
2
,Σ
2
,L
2
,→
2
are defined by the following rules:
t=q
e(α),l
−−−→q
′
q
e(α),l∪{”mutable”}
−−−−−−−−−−−→q
′
t
2
̸=t
t
2
We are now ready to present our test case mutation
algorithm given in Algorithm 1: it takes a mutation
operator M along with a test case set TC. It produces
Algorithm 1: IOTS Test Case Mutation.
input : Test case set TC, Mutation Operator M
output: Test case set TC
M
1 TC
M
:=
/
0;
2 foreach tc ∈ TC do
3 foreach q
e(α),l
−−−→ q
′
∈ mutable(M) in ts such that
ts = q0
(e
1
(α
1
),l
1
)...(e
k
(α
k
),l
k
)
−−−−−−−−−−−−−→ pass ∈ tc and
selection(TC, TC
M
) do
4 mark q
e(α),l
−−−→ q
′
in tc such that q
e(α),l
−−−→ q
′
∈ mutable(M)in ts arbitrarly chosen;
5 tc
2
:= M.Change(tc,q
e(α),l
−−−→ q
′
);
6 tc
2
:= M.Expected(tc
2
);
7 compl tc
2
;
8 TC
M
:= TC
M
∪ {tc
2
};
a new test case set, denoted TC
M
. It covers every mu-
table test step of a test sequence ts (line 3) starting
from the initial state of the test case such that ts is
finished by the state pass. We choose to only mu-
tate test sequences finished by pass to avoid bringing
confusion in the test result analysis. Indeed, if we
mutate a test sequence finished by fail and if we ob-
tain a fail verdict while testing, it is very difficult to
deduce whether SU T is faulty on account of the mu-
tation. As the set of mutants may become large, Algo-
rithm 1 calls the function selection(TC,TC
M
), which
returns a boolean value. This function expresses a
mutant generation strategy, e.g., ”applies M on every
test case only once”, which stops the mutation of the
test cases once some conditions are met. In this case,
the function returns false. Algorithm 1 marks the cho-
sen test step with ”mutable” to help the mutation op-
erator target the test step to change. A new test case
tc
2
is built by applying the function M.Change and by
completing its branches not finished by a verdict state
with M.Expected in order to express the expected be-
haviour after the execution of the mutated test step.
Additionally, the mutant tc
2
is completed once more
(line 7) with the operator compl : IOT S → IOT S to
add transitions that express all the behaviours that
might be observed and the related test verdicts. The
resulting mutant tc
2
is stored in TC
M
. The operator
compl is defined by:
Definition 5 (IOTS Operator compl) : compl tc =
⟨Q
2
,q0,Σ
2
∪ {θ},L,→
2
⟩ is the IOTS test case ob-
tained from tc where Q
2
,Σ
2
,→
2
are defined by the
following rules:
r
1
:q
1
e(α),l
−−−→ q
2
⊢ q
1
e(α),l
−−−→ q
2
r
2
:q
1
e(α),l
−−−→ q
2
,q
1
!∗,{}
−−−→ q
3
/∈→⊢ q
1
!∗,{}
−−−→ inc
r
3
:q
1
!e(α),l
−−−−→ q
2
,q
1
?e
2
(α
2
),l
−−−−−→ q
3
/∈→,q
1
θ
−→ q
3
/∈→⊢
q
1
θ
−→ f ail
ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering
586

The inference rule r
1
takes all the transitions of an
IOTS to build a new test case. r
2
completes the test
case with a new transition to express that any unex-
pected output leads to the inconclusive verdict. When
the test case only expects outgoing transitions labelled
by output events, the rule r
3
also adds a transition to
fail modelling that the absence of reaction is faulty.
The function selection(TC,TC
M
) encodes con-
ditions on the test case sets TC and TC
M
to limit
the number of mutants by mutation operator. Vari-
ous conditions and combinations could be considered.
Here, we provide some examples:
• No restriction (all mutable test steps are covered);
• Every test case is mutated at most n times
• Every mutable test step of each test case is mu-
tated at most n times
Figure 4: IOTS Mutated Test Case example.
Figure 4 illustrates an example of mutant obtained
from the test case of Figure 2 by applying the opera-
tor ”Token removal” on the second mutable test step
(!/ok). The verdict is pass if a response is observed
with a status 403 or a status 401, which encodes that
the request has been rejected on account of insuffi-
cient permissions.
3.3 Generation of Concrete Test Cases
Finally, executable test scripts are generated from
IOTS test cases. We have chosen to generate test
cases using the frameworks Citrus and Mockserver.
Given an IOTS test case tc ∈ TC
M
, some parame-
ters may still be assigned to ”*”. For example, this
happens for parameters used to identify sessions. In
short, we assign these parameters with stored values
collected from Log files or with random values. To
generate a test script, the transitions of tc labelled by
”mock” are initially pruned. The resulting IOTS is
converted as follows: every request to SU T is con-
verted into code that calls SU T and waits for a re-
sponse. An example is given in Figure 5. The transi-
tions labelled by responses of this request are used to
build assertions. The test script ends with the call of
the method ”verificationMock”, which aims to check
whether mock components behave as expected during
the test execution. At the moment, we check whether
the number of calls to a mocked request matches with
the number of time this request is found in tc.
To generate mock components, the IOTS transi-
tions of tc labelled by ”mock” are used to derive
rules of the form request()...respond(), which mimic
the behaviour of a dependee service. Then, the
method ”verificationMock” is written according to
these rules. Figure 6 shows a rule example written
with the language provided by the framework Mock-
Server.
@Test @CitrusTest
2 public void testAccMan() throws FileNotFoundException{
HttpClient toClient = CitrusEndpoints
4 . http () . client () . requestUrl (” http :// AccMan/”).build() ;
$(HTTP()
6 . client ( toClient ) .send() . get (”checkAccountRisk”).message()
.header(”token”,1234) . body(”\”acc\”=99”)
8 . accept (MediaType.ALL VALUE));
$( receive ( toClient )
10 .message(). type (MessageType.PLAINTEXT).name(”Response”)
. extract (fromHeaders()
12 . header(HttpMessageHeaders.HTTP STATUS CODE,
”statusCode”))
.header(”token ”,” token”) ) ) ;
14 variable (”body”,” citrus :message(Response.body())”) ;
variable (” status ”, ”${statusCode}”);
16 String status = context . getVariable (” status ”) ;
String t = context . getVariable (”token”) ;
18 If (token. equals (”1234”) && status. equals (”403”))
assertTrue ( true ) ;
else Assumptions.assumeTrue(false ,” Inconclusive ”) ;
20 verificationMock () ;}
Figure 5: Example of test script for the service AccMan.
mockServer.when(
2 request () .withMethod(”GET”).withPath(”/evaluateRisk”)
.withHeaders( new Header(”acc”, ”99”) ,new Header(”token”,
”1234”))
4 ,Times.exactly (1) )
.respond( response () .withStatusCode(200)
6 .withBody(”LOWRISK”));
Figure 6: Mock component piece of code, which imple-
ments the events !/EvaluateRisk and !ok of the test case of
Figure 4.
4 CONCLUSION
In this paper, we present an original solution to gener-
ate mutated test cases for testing the security of REST-
ful APIs. We propose a list of 18 mutation operators
specialised in the generation of security test cases.
Subsequently, we introduced an algorithm allowing to
generate concrete test scripts and mock components
by means of these operators. A significant contribu-
Security Testing of RESTful APIs with Test Case Mutation
587

tion of this algorithm is its ability to generate mock
components to test a RESTful API in isolation. We
provide a preliminary evaluation of our algorithm to
study the effectiveness of the mutated test cases and
how its efficiency. This part is detailed in (Salva and
Sue, 2024). Our results demonstrate its capability to
construct hundreds of test cases and mock compo-
nents within minutes, and show good scalability. Be-
sides, the mutants enable the detection of weaknesses
in REST APIs and enhance code coverage.
At the moment, our mutation operators allow to
infer mutants that mimic attacks performed by one
test step. As part of future work, we aim to define
more sophisticated operators that could support the
mutation of several steps at a time, thus constructing
more complex attack scenarios.
ACKNOWLEDGMENT
Research supported by the industrial chair
on Digital Confidence https://www.uca-
fondation.fr/chaires/confiance-numerique/
REFERENCES
Abu-Dabaseh, F. and Alshammari, E. (2018). Automated
penetration testing: An overview. In The 4th Interna-
tional Conference on Natural Language Computing,
Copenhagen, Denmark, pages 121–129.
Arcuri, A. (2018). Test suite generation with the many in-
dependent objective (mio) algorithm. Information and
Software Technology, 104:195–206.
Arcuri, A. (2019). Restful api automated test case gen-
eration with evomaster. ACM Trans. Softw. Eng.
Methodol., 28(1).
Arcuri, A., Iqbal, M. Z., and Briand, L. (2011). Ran-
dom testing: Theoretical results and practical impli-
cations. IEEE transactions on Software Engineering,
38(2):258–277.
CAPEC (2024). Common attack pattern enumeration and
classification, https://capec.mitre.org/.
CWE (2024). Common weakness enumeration,
https://cwe.mitre.org/.
Ferreira, J. and Paiva, A. C. R. (2019). Android testing
crawler. In Piattini, M., da Cunha, P. R., de Guzm
´
an, I.
G. R., and P
´
erez-Castillo, R., editors, Quality of Infor-
mation and Communications Technology - 12th Inter-
national Conference, QUATIC, Ciudad Real, Spain,
volume 1010 of Communications in Computer and In-
formation Science, pages 313–326. Springer.
K
¨
oroglu, Y. and Sen, A. (2018). TCM: Test Case Muta-
tion to Improve Crash Detection in Android. In Pro-
ceedings of the 21st International Conference on Fun-
damental Approaches to Software Engineering, pages
264–280. Springer.
Li, W., Le Gall, F., and Spaseski, N. (2018). A survey on
model-based testing tools for test case generation. In
Itsykson, V., Scedrov, A., and Zakharov, V., editors,
Tools and Methods of Program Analysis, pages 77–
89, Cham. Springer International Publishing.
Paiva, A., Restivo, A., and Almeida, S. (2020). Test case
generation based on mutations over user execution
traces. Software Quality Journal, 28.
Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Le Traon, Y.,
and Harman, M. (2019). Mutation testing advances:
an analysis and survey. In Advances in Computers,
volume 112, pages 275–378. Elsevier.
Phillips, I. C. C. (1987). Refusal testing. Theor. Comput.
Sci., 50:241–284.
Salva, S. and Sue, J. (2023). Automated test case gen-
eration for service composition from event logs. In
38th IEEE/ACM International Conference on Auto-
mated Software Engineering, ASE 2023 - Workshops,
Luxembourg, September 11-15, 2023, pages 127–134.
IEEE.
Salva, S. and Sue, J. (2024). Security testing of
restful apis with test case mutation, full paper.
https://arxiv.org/abs/2403.03701.
Salva, S. and Zafimiharisoa, S. R. (2014). Model reverse-
engineering of Mobile applications with exploration
strategies. In Ninth International Conference on
Software Engineering Advances, ICSEA 2014, Nice,
France.
Skouloudi, C., Malatras, A., Naydenov, R., and Dede, G.
(2018). Good practices for security of internet of
things in the context of smart manufacturing testing.
Sue, J. and Salva, S. (2024). Security testing of
restful apis with test case mutation, companion
site. https://github.com/JarodSue/Restful-API-test-
case-mutation.
Tretmans, J. (2008). Model Based Testing with Labelled
Transition Systems, pages 1–38. Springer Berlin Hei-
delberg, Berlin, Heidelberg.
Xu, Z., Kim, Y., Kim, M., Rothermel, G., and Cohen, M. B.
(2010). Directed test suite augmentation: Techniques
and tradeoffs. FSE ’10, page 257–266, New York, NY,
USA. Association for Computing Machinery.
Xuan, J., Xie, X., and Monperrus, M. (2015). Crash repro-
duction via test case mutation: Let existing test cases
help. In Proceedings of the 2015 10th Joint Meeting
on Foundations of Software Engineering, ESEC/FSE
2015, page 910–913, New York, NY, USA.
ENASE 2024 - 19th International Conference on Evaluation of Novel Approaches to Software Engineering
588

APPENDIX
Table 1: Test case mutation operators for the security testing of RESTful APIs.
Mutation Description Expected Behaviour Mutation Condition
Event Duplication duplicate a request event to SU T q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧ f rom(q
!∗(α),l
−−−→ pass) =
SUT
e == ∗
HTTP Verb Change changing the HTTP verb of a re-
quest
q
!∗(α),l
−−−→ pass with status(α) := 405 ∧ f rom(q
!∗(α),l
−−−→
pass) = SUT
isReq(e(α))==true
XSS attack XSS attack q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧
contains(”error”,body(α)) ∧ f rom(q
!∗(α),l
−−−→ pass) = SUT
body(α)! =
”” ∨ header(α)! = ””
Cryptographic failures Replay an event using untrusted
connexion
q
!∗(α),l
−−−→ pass with contains(”ERR CERT AUT HORITY
INVALID”, body(α)) ∧ f rom(q
!∗(α),l
−−−→ pass) = SUT
e == ” ∗ ”
Token Removal Delete a token in event q
!∗(α),l
−−−→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
”token” ∈ l
Token Removal on the
creation
Delete a token in event q
!∗(α),l
−−−→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
”token creation” ∈ l
Token Alteration Replacing a token by another one;
three types: expired authentication
token, token existing but not for this
session, and token not existing
q
!∗(α),l
−−−→ pass with status(α) ≥ 401 ∧ status(α) ≤ 403 ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
”token” ∈ l
Stress Testing replay events to the tested service a
lot of times in a small window
q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧
¬contains(”error”body(α))∧ f rom(q
!∗(α),l
−−−→ pass) = SUT
e == ” ∗ ”
SSRF Enforce “deny by
default”
request or response from an un-
known service
q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧
(contains(”error”,body(α)) ∨ status(α) := 404) ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
e == ” ∗ ”
Body data manipulation replay request using unauthorized
data
q
!∗(α),l
−−−→ pass with (status(α) := 400 ∨ status(α) := 422) ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
body(α)! =
”” ∨ header(α)! = ””
Cookie manipulation change a cookie to inject an attack q
!∗(α),l
−−−→ pass with status(α) := 400 ∧ f rom(q
!∗(α),l
−−−→
pass) = SUT
cookies(α)! = ””
Failed Login Attempt
Duplication
duplicating login event with wrong
credentials
q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧ contains(”error :
TooManyFailedAttempt”,body(α)) ∧ f rom(q
!∗(α),l
−−−→
pass) = SUT
isReq(e(α)) ∧
”login” ∈ l
Path manipulation change URL to get unauthorised ac-
cess to data
q
!∗(α),l
−−−→ pass with status(α) := 404 ∧ f rom(q
!∗(α),l
−−−→
pass) = SUT
isReq(e(α))
SQL injection manipulate input data to inject SQL
code
q
!∗(α),l
−−−→ pass with (status(α) := 400 ∨
contains(”error”,body(α))) ∧ f rom(q
!∗(α),l
−−−→ pass) = SUT
body(α)! = ””
Session management add a (long) delay during which no
reaction should be observed before
the next event
q
!∗(α),l
−−−→ pass with status(α) := 401 ∧ contains(”error :
sessionterminated”body(α)) ∧ f rom(q
!∗(α),l
−−−→ pass) = SUT
e == ” ∗ ”
Information leakage modify a request to get access to
sensitive information
q
!∗(α),l
−−−→ pass with status(α) := 401 ∧ ”crash” /∈ l ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
isReq(e(α))
Dependee service shut-
down
shutdown a mock component after
requesting it
q
!∗(α),l
−−−→ pass with ”crash” /∈ l ∧ (contains(”error :
connexiontimedout”,body(α)) ∨ status(α) := 408) ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
q
∗(α),l
−−−→ with
”mock” ∈ l
Buffer overflow overflow input data for trying to
crash a server
q
!∗(α),l
−−−→ pass with status(α) := 400 ∧ ”crash /∈ l ∧
f rom(q
!∗(α),l
−−−→ pass) = SUT
e == ” ∗ ”
Security Testing of RESTful APIs with Test Case Mutation
589