Mutation Operators for Mutation Testing of Angular Web Applications

Sarah Augustin, Hendrik Winkelmann

and Herbert Kuchen

Department of Information Systems, University of M

unster, Leonardo-Campus 3, M

unster, Germany

Keywords:

Mutation Testing, Mutation Operators, Angular, Web Applications.

Abstract:

Mutation testing is an approach for assessing the quality of a test suite by using mutation operators to insert

changes into the code and then checking whether the test suite can detect the inserted changes. Due to the

growing prevalence and complexity of web applications, the importance of web testing has increased, making

mutation testing a potentially beneﬁcial approach for web applications. Since in web applications, mostly

web-speciﬁc mistakes and not generic mistakes occur, the question arises, to whether new mutation operators

simulating such realistic, web-speciﬁc mistakes perform better than the traditional, generic mutation operators.

The work at hand addresses this question by developing new mutation operators speciﬁc to the client-side

TypeScript code of Angular web applications and evaluating how they perform in comparison to the traditional

mutation operators. The ﬁndings indicate that the new web-speciﬁc mutation operators introduce fewer, more

realistic, and harder-to-kill mutants than the traditional mutation operators, thus being a promising approach

for assessing the test suite quality of web applications.

1 INTRODUCTION

Since the 1970s, mutation testing has gained increas-

ing attention from both academia and industry, lead-

ing to, e.g., its integration into Google’s mandatory

code review process (Petrovi

c and Ivankovi

c, 2018;

Jia and Harman, 2011). Mutation testing can be ap-

plied to a variety of different types of applications in-

cluding web applications. Applying mutation testing

to web applications can contribute to ensuring a high

quality of web tests. Since, according to Marchetto

et al., almost three-quarters of the bugs in web appli-

cations are web-speciﬁc, the question arises whether

mutation testing can be applied to web applications

with its traditional mutation operators or whether new

mutation operators speciﬁcally targeting such web-

speciﬁc faults would perform better (Marchetto et al.,

2009).

The work at hand addresses this question by devel-

oping new mutation operators inserting web-speciﬁc

faults into the application code during mutation test-

ing. More speciﬁcally, eight new mutation opera-

tors for mutating code related to comparisons, error

handling, input default values, input validation, input

names, subscribe calls, unsubscribe calls, and RxJS

operators were introduced. Those new mutation op-

erators are compared to the traditional mutation op-

erators in order to assess whether inserting realistic,

https://orcid.org/0000-0002-7208-7411

https://orcid.org/0000-0002-6057-3551

web-speciﬁc faults provides beneﬁts compared to in-

serting the traditional, generic faults. Since it can be

argued that faults depend on the web framework used,

this work speciﬁcally considers web applications built

with the Angular framework. Angular was chosen

because it is one of the most popular web frame-

works (Malcher et al., 2023). Following the ﬁnding

of Ocariza et al. that the majority of faults arise nei-

ther in the server-side code nor in the HTML code, but

instead in the client-side code, this work speciﬁcally

focuses on the client-side TypeScript code of Angular

web applications (Ocariza et al., 2013, p. 56).

This work contributes to theory by providing

proof by construction that creating mutation operators

simulating realistic mistakes is a promising approach

worthy of being researched in depth. The contribu-

tion towards practice corresponds to providing soft-

ware developers with new mutation operators for mu-

tation testing of Angular web applications. If used

instead of the traditional mutation operators, the new

mutation operators contribute to speeding up the pro-

cess of mutation testing and reducing the human effort

needed to check the output of mutation testing.

This work is organized as follows. While Section

2 describes the research method, an overview of the

theoretical background is provided in Section 3. In

turn, Section 4 the design of the new mutation opera-

tors is illustrated. The implementation and evaluation

are presented in Section 5 and Section 6. The work

ﬁnishes with a conclusion in Section 7.

390

Augustin, S., Winkelmann, H. and Kuchen, H.

Mutation Operators for Mutation Testing of Angular Web Applications.

DOI: 10.5220/0013203900003928

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2025), pages 390-397

ISBN: 978-989-758-742-9; ISSN: 2184-4895

2 RESEARCH METHOD

For the design of new mutation operators simulating

realistic programming mistakes within the TypeScript

ﬁles of Angular web applications, ﬁrst, those realistic

mistakes need to be identiﬁed. To identify the mis-

takes, two research methods were chosen: a system-

atic literature review and semi-structured expert inter-

views.

The systematic literature review was conducted

according to the work of vom Brocke et al. and in-

cluded searching the database Scopus with a search

string and extending the results with a forward and

backward search (vom Brocke et al., 2009). To

enrich the results of the literature review, addition-

ally, four semi-structured expert interviews were con-

ducted. All four interviewees are software developers

working full-time in the IT industry and have devel-

oped Angular web applications professionally. One

of the interviewees also gives professional training

courses on the topic of Angular web applications. The

programming mistakes identiﬁed through the system-

atic literature review and the interviews were grouped

into categories.

To narrow down the scope, some categories were

selected to be focused upon for the design of the

new mutation operators. The categories were chosen

based on the criteria that they should be applicable to

all Angular web applications, that they should have

been included in at least two sources, and that at least

one of the two sources had to be an interview. These

criteria aim at selecting categories of high relevance

by ensuring that they have occurred in several sources.

The decision to require that the category must have at

least come up in one interview was made because the

interviews, in contrast to the papers, were speciﬁc to

Angular rather than general for all web applications.

For each of the mistakes in the remaining categories,

a new mutation operator was designed that simulates

it.

3 THEORETICAL BACKGROUND

AND RELATED WORK

First, we explain mutation testing in detail in Subsec-

tion 3.1. Afterward, in Subsection 3.2, more details

about advances and research in the ﬁeld of mutation

testing are presented.

3.1 Traditional Mutation Testing

Mutation testing is an approach used to assess the

quality of a test suite. When applying mutation test-

ing to a program, mutants are automatically created

for that program (Offutt and Untch, 2001). A mutant

is a version of the original program in which a small

change was made (Offutt and Untch, 2001). Those

small changes are inserted by applying transforma-

tion rules called mutation operators (Mike Papadakis

et al., 2019; Jia and Harman, 2011). The test suite can

be run on each mutant to check whether at least one

test fails. If so, the mutant is called dead or killed, oth-

erwise, it is called alive or a surviving mutant (Offutt

and Untch, 2001). Equivalent mutants which corre-

spond to functionally equivalent versions of the orig-

inal program are removed from the group of the sur-

viving mutants (Mike Papadakis et al., 2019). The

remaining surviving mutants point out inadequacies

of the test suite insofar as the inserted defect was not

detected (Offutt and Untch, 2001). The test suite can

be improved by adding new test cases killing these

mutants (Offutt and Untch, 2001).

3.2 Advances and Research in Mutation

Testing

Research in the ﬁeld of mutation testing has cov-

ered various aspects. One research approach aim-

ing at inserting more realistic and complex faults is

the “higher order mutation testing” in which muta-

tion operators are applied several times to create a

mutant (Harman et al., 2010). The disadvantage of

higher-order mutation testing is the immense num-

ber of possible fault combinations, which can make

it too computationally expensive and impractical (Jia

and Harman, 2009; Harman et al., 2010). The ap-

proach of higher-order mutation testing is similar to

the approach of this work insofar as both aim at in-

serting more realistic faults. This work circumvents

the disadvantage of the search space explosion by us-

ing domain knowledge. Instead of all combinations of

ﬁrst-order faults, only realistic faults are considered.

Another research ﬁeld is the reduction of the com-

putational cost of mutation testing as this is one of its

main problems (Jia and Harman, 2011). According

to Offutt and Untch, research on reducing those com-

putational expenses can typically be categorized into

one of the three strategies of “do fewer, do smarter,

or do faster” (Offutt and Untch, 2001, p. 37). The

idea of the “do fewer” approach is to reduce the num-

ber of mutants without an intolerable information loss

(Offutt and Untch, 2001). One approach in literature

for reducing the number of mutants is, for example,

selective mutation, with which some mutation opera-

tors are selected according to a certain criterion and

only those mutation operators are used to generate

mutants (Mike Papadakis et al., 2019; Jia and Har-

Mutation Operators for Mutation Testing of Angular Web Applications

391

man, 2011). The category of “do fewer” strategies

can also contribute to mitigating human oracle prob-

lem which corresponds to the high amount of human

effort needed to check the output of the mutation test-

ing (Jia and Harman, 2011). If, with the “do fewer”

approach, fewer mutants are generated, it can be ar-

gued that checking the output of the mutation testing

takes less effort. This work could be classiﬁed as a

“do fewer” approach if the new mutation operators

are used instead of the existing mutation operators, as

they generate fewer mutants.

Research has also addressed the development of

new mutation operators for various scenarios. Some

researchers have introduced mutation operators for in-

serting a speciﬁc type of fault into the code (Mike

Papadakis et al., 2019). To the best of the authors’

knowledge, however, no research has yet focused on

developing a set of web-speciﬁc mutation operators

for client-side TypeScript code.

4 DESIGN OF THE MUTATION

OPERATORS

Following the research method described in Sec-

tion 2, a systematic literature review and four semi-

structured expert interviews were conducted. The lit-

erature review identiﬁed 39 different programming

mistakes grouped into 15 categories, while the expert

interviews identiﬁed 20 different programming mis-

takes grouped into 11 categories. When combining

those mistakes, a total of 55 mistakes in 23 categories

was identiﬁed. The number does not correspond to

the sum since some mistakes were identiﬁed in both

the literature review and the interviews. Applying the

criteria described in Section 2, four categories were

selected to focus upon: comparisons, error handling,

forms, and RxJS. RxJS is a library for reactive pro-

gramming commonly used in Angular (Malcher et al.,

2023). Those four categories include a total of eight

mistakes. For each mistake, a new mutation opera-

tor was designed that simulates it. To the best of the

authors’ knowledge, the new mutation operators are

all novel, as no web-speciﬁc mutation operators for

client-side TypeScript code operators have been intro-

duced in literature yet. In the following subsections,

the new mutation operators are described in detail.

4.1 Comparison Mutation Operator

The ﬁrst new mutation operator called “comparison

mutation operator” simulates the programming mis-

take of using a loose comparison operator instead

of a strict one or vice versa. The loose operators

“==” and “!=” perform ﬁrst an automatic type con-

version if the left- and right-hand sides are of a differ-

ent data type and then compare the values for equal-

ity (Mozilla, 2023). The strict operators “===” and

“!==” in contrast do not perform a type conversion

(Mozilla, 2023). The new comparison mutation op-

erator simulates the mistake of confusing them by

switching “==” to “===”, “!=” to “!==”, “===” to

“==”, and “!==” to “!=” in the TypeScript code. Test

cases which depend on the comparisons always re-

sulting in the expected boolean values can, for exam-

ple, be used to kill such mutants.

4.2 Error Handling Mutation Operator

The second new mutation operator is called “error

handling mutation operator”. It simulates the mis-

take of not catching and handling an exception. In the

TypeScript ﬁles of Angular web applications, errors

can occur both synchronously and asynchronously.

Asynchrony in Angular is typically handled by either

the datatype Observable or the datatype Promise.

Depending on where the error occurs, error handling

can take place in four different ways:

1. catching synchronous errors in a try-catch block

2. catching asynchronous errors in a Promise

3. catching asynchronous errors in the pipe method

of an Observable

4. catching asynchronous errors in the subscribe

block of an Observable

The new error handling mutation operator deletes

the code responsible for catching and handling such

errors. The mutants generated by this mutation oper-

ator can, for example, be killed by test cases which

purposefully trigger those errors and test whether the

application handles those errors properly.

4.3 Input Default Value Mutation

Operator

The third new mutation operator is called “input de-

fault value mutation operator”. It simulates the mis-

take of setting an incorrect default value for a form

ﬁeld. This is done by mutating the default values of

form ﬁelds. If the default value is an empty string, it

is replaced by ’mutated string’. If it is a non-empty

string it is replaced by an empty string. If the default

value is a numeric literal, its value is increased by one,

and if it is a boolean literal, it is inverted. In Angular,

the most common type of form is the “reactive form”

(Malcher et al., 2023). A reactive form, including its

ENASE 2025 - 20th International Conference on Evaluation of Novel Approaches to Software Engineering

392

default values, can be deﬁned in various ways. The in-

put default value mutation operator supports all those

various ways. The mutants generated by this muta-

tion operator can, for example, be killed by test cases

using the default value of a form ﬁeld instead of an

individually entered value.

4.4 Input Validation Mutation Operator

The fourth new mutation operator is called “input val-

idation mutation operator”. It simulates the mistake

of forgetting to set up input validation for a form ﬁeld

or an entire form. The mutation operator thus deletes

the client-side input validation which ensures that the

user has entered valid data into the form before sub-

mitting it. In general, several ways of deﬁning input

validation for reactive forms exist in Angular. Inde-

pendent of which way is used, the new input vali-

dation mutation operator deletes the input validation.

The mutants generated by this mutation operator can,

for example, be killed by test cases purposefully in-

serting invalid data into a form and checking whether

this case is handled properly. It thus encourages writ-

ing test cases representing unexpected user behavior.

4.5 Input Name Mutation Operator

The ﬁfth new mutation operator is called “input name

mutation operator”. It simulates the mistake of as-

signing an incorrect name to a form ﬁeld. The in-

put name mutation operator thus changes the name of

a form ﬁeld by appending the postﬁx “ mutated” to

it. As explained in further detail in Section 5, due

to technical reasons, it was not possible to mutate

names that were deﬁned as class properties. All other

ways of how names can be set are supported by the

new mutation operator. This mutation operator en-

courages writing test cases checking whether all in-

put ﬁelds work as expected and process their values

correctly, which would not be possible if an incorrect

name would be assigned.

4.6 Subscribe Call Mutation Operator

The sixth new mutation operator, which is called

“subscribe call mutation operator”, simulates the mis-

take of forgetting to subscribe to an Observable.

Since an Observable is lazy and is only invoked if

it is being subscribed to, forgetting the subscription

has the effect that the Observable is never executed

(Lesh et al., 2022). In Angular, an Observable is

subscribed by calling the subscribe method on it.

The new mutation operator deletes the subscribe call

including everything that is passed to the method.

Mutants generated by this mutation operator can be

killed by adding test cases which depend on the

Observable to actually be invoked.

4.7 Unsubscribe Call Mutation

Operator

The seventh new mutation operator is called “un-

subscribe call mutation operator”. It simulates the

mistake of forgetting to unsubscribe an Observable.

Unsubscribing is necessary if the Observable never

completes on its own and if its execution is thus inﬁ-

nite (Lesh et al., 2022). Without unsubscribing, com-

putation power and memory resources are wasted, po-

tentially causing memory leaks (Lesh et al., 2022).

Two ways of unsubscribing from an Observable

exist. First, the unsubscribe() method can be called

on an existing subscription (Malcher et al., 2023).

Second, when creating an Observable, the pipe()

method can be called on it. To the pipe method, sev-

eral RxJS operators can be passed which manipulate

the emitted stream of values. The ﬁve RxJS opera-

tors first(), take(), takeUntil(), takeWhile(),

and takeUntilDestroyed() can be used to automat-

ically unsubscribe from the Observable, e.g., based

on a certain condition passed to the operator (Malcher

et al., 2023). The new mutation operator deletes both

the unsubscribe calls and the ﬁve listed RxJS opera-

tors from the TypeScript ﬁles of the Angular web ap-

plication. This mutator encourages writing test cases

which depend on the Observable to be ﬁnished as

required.

4.8 RxJS Mutation Operator

The eighth and last new mutation operator is the

“RxJS mutation operator”. It simulates the mistake of

using the wrong RxJS operator. According to the of-

ﬁcial RxJS website, over one hundred operators such

as map(), reduce(), and switchMap() exist, each

of which can be applied to the values emitted by an

Observable (Lesh et al., 2023). The new RxJS mu-

tation operator switches the RxJS operators within the

TypeScript code to another one. As explained in fur-

ther detail in Section 5, due to technical reasons, it is

not possible to switch an RxJS operator to an arbitrary

other RxJS operator, but only to another RxJS opera-

tor which has already been imported into the ﬁle. The

RxJS operators are thus switched with the ﬁrst im-

ported operator that has a different name than itself.

The mutants generated by this mutation operator can

be killed by test cases which depend on the values of

the Observable to be changed as required by the con-

text.

Mutation Operators for Mutation Testing of Angular Web Applications

393

5 IMPLEMENTATION OF THE

MUTATION OPERATORS

For the implementation of the new mutation opera-

tors, the existing mutation testing tool StrykerJS was

taken as a starting point and was extended by the

new mutation operators.

StrykerJS was selected be-

cause it is the most common mutation testing tool for

JavaScript, because it is an open-source project, and

because it already offers sixteen built-in mutation op-

erators to which the new mutation operators can be

compared (S

anchez et al., 2022; Info Support, 2023).

StrykerJS performs the mutations not in the Type-

Script code directly but in the abstract syntax tree

(AST) of the TypeScript code. When making a muta-

tion, StrykerJS replaces one node in the AST by an-

other node. This new node can, e.g., be created com-

pletely new or can be created as a copy of the existing

node and then be modiﬁed. The new AST node is then

transformed back into TypeScript code by StrykerJS.

Due to performance reasons, StrykerJS uses an ap-

proach called mutation switching for conducting mu-

tation testing (Jansen, 2020). With mutation switch-

ing, all mutants are inserted into the same ﬁle (Jansen,

2020). By using, e.g., if-else-constructs, the mutants

can be activated and deactivated in that ﬁle, so that

for each run of the test suite only one mutant is ac-

tive (Jansen, 2020). This has a performance advan-

tage since only this one ﬁle needs to be transpiled and

compiled (Jansen, 2020). One disadvantage, however,

is that it allows only mutations of AST nodes, which

are located at places where if-else-constructs are al-

lowed. Due to this reason, the RxJS mutation operator

cannot switch an RxJS operator to an non-imported

other RxJS operator, since the import statement can-

not be changed. Similarly, this is also the reason for

why the input name mutation operator cannot mutate

names that are deﬁned as class properties.

6 EVALUATION OF THE

MUTATION OPERATORS

For the evaluation, the new mutation operators were

applied to an existing Angular web application. This

web application is a desk sharing tool developed com-

mercially. For unit testing of the web application,

the testing framework Jest is used. Currently, 24 test

suites with a total of 138 tests exist.

Mutation testing was applied with three different

sets of mutation operators to this web application:

with only the eight new mutation operators, with only

https://stryker-mutator.io/docs/stryker-js/introduction/

the 16 mutation operators which were already pre-

implemented in StrykerJS, and with all 24 mutation

operators in combination. In the following subsec-

tions, ﬁrst, the result of the mutation testing is pre-

sented. Then the execution time, the realism of the

mistakes, and expert feedback are discussed in fur-

ther detail. Afterward, the ﬁndings are combined for

a ﬁnal discussion.

6.1 Mutation Testing Results

For all three versions – for applying only the new,

only the traditional, or all mutation operators – the

distribution of the mutants on the four different sta-

tuses killed, survived, no coverage, and timeout was

determined. No coverage means that the mutant sur-

vived because it was not even covered by any test.

Timeout means that StrykerJS has stopped the exe-

cution due to it taking unexpectedly long. This can,

for example, happen if accidentally an inﬁnite loop

is created by the change made by the mutation oper-

ator. The two further statuses of compile errors and

runtime errors are not considered, since they did not

occur. The distribution in percent can be seen in Table

1. In absolute numbers, the eight new mutation oper-

ators generated a total number of 192 mutants, while

the 16 existing, traditional mutation operators gener-

ated a total 1114 mutants.

The mutation testing result of the eight new muta-

tion operators can be beneﬁcial in several ways. First

of all, the results can be used to improve the quality

of the test suites. For example, for the 58,85% of mu-

tants that survived, it could be checked which tests

have covered them and why they did not detect the

inserted mistake. Those tests can then be improved

so that they afterwards catch the inserted mistakes,

thus ensuring that if such mistakes are accidentally in-

serted in the future, they will be caught. Similarly, the

21,35% of mutants that were not covered can be used

to get insights into which parts of the code are not

covered by tests yet. This information can motivate

the developer to write additional tests covering that

part of the code. The mutation testing result might

additionally contribute to ﬁnding mistakes by direct-

ing the programmer’s attention to the critical parts of

Table 1: Status distribution of the mutants per mutation op-

erator type in percent.

Mutation

operators

Killed Sur-

vived

cover-

age

Time-

out

New 19,79% 58,85% 21,35% 0,00%

Traditional 29,17% 33,30% 37,16% 0,36%

All 27,79% 37,06% 34,84% 0,31%

ENASE 2025 - 20th International Conference on Evaluation of Novel Approaches to Software Engineering

394

the code in which common mistakes occur.

Comparing the mutation testing results of the new

mutation operators to the existing, traditional muta-

tion operators, one can see that 19.79% of the mu-

tants generated by the new mutation operators were

killed, while in contrast 29.17% of the mutants gener-

ated by the traditional mutation operators were killed.

That indicates that the new mutation operators gener-

ate harder-to-kill mutants, which therefore potentially

provide more value and insights to the programmer.

Looking at the column entitled “survived”, one

can see that 58.85% of the mutants generated by

new mutation operators survived, while only 33.30%

of the traditional mutation operators survived. This

shows that the new mutation operators generate a

higher percentage of mutants that are covered by a

test but that are still not killed. This might indicate

that the new mutation operators have a higher poten-

tial for improving the existing test cases that cover

those survived mutants. The programmer can check

which test case covered the mutant and can improve

it to catch the mistake inserted by the mutation oper-

ator.

In the next column, one can see that 21.35% of

the mutants generated by the new mutation operators

were not covered while 37.16% of the mutants gen-

erated by the traditional mutation operators were not

covered. This might indicate two things. First, the

new mutation operators seem to generate more mu-

tants in areas of code already covered by tests than

the traditional mutation operators, conﬁrming that the

tests address relevant code areas which often include

mistakes. Second, the traditional mutation opera-

tors seem to perform better at highlighting uncovered

parts of the code than the new mutation operators. To-

wards that goal, however, other metrics such as code

coverage can be used instead of mutation testing.

All in all, it becomes clear that the traditional mu-

tation operators highlight more areas of the code that

are not covered yet, thus motivating the programmer

to write more tests for the uncovered parts of the

code. At the same time, the new mutation opera-

tors insert more changes at locations already covered

by test cases than the traditional mutation operators

but still achieve a lower percentage of killed mutants.

This shows that they insert hard-to-kill changes that

require the existing test cases to be improved.

6.2 Execution Time

Mutation testing was executed on the web application

in three versions: with only the eight new mutation

operators, with only the 16 traditional mutation oper-

ators, and with all 24 mutation operators. As was al-

Table 2: The execution time of mutation testing.

Mutation operators Average execution time

New 21min

Traditional 92min

All 112min

ready mentioned in the previous subsection, the new

mutation operators inserted 192 mutants. This makes

an average of 24 mutants per new mutation opera-

tor. The traditional mutation operators inserted 1114

mutants, corresponding to an average of around 69

mutants per traditional mutation operator. Since the

number of mutants is lower for the new mutation op-

erators, this indicates that using the new mutation op-

erators instead of the traditional mutation operators

might be beneﬁcial in terms of execution time.

The execution time was determined for the three

versions of using only the new, only the traditional,

and all mutation operators. Since the execution time

is inﬂuenced by the timeout setting of StrykerJS, the

timeout was set to 30 seconds across all runs. Since

the execution time might vary per run, four runs were

conducted, and the average was calculated. The re-

sults rounded to minutes can be seen in Table 2. As

can be seen in the table, the new mutation operators

have a lower execution time than the traditional mu-

tation operators.

6.3 Past Mistakes

To evaluate the new mutation operators, it was

checked whether the mistakes inserted by them have

already occurred in the web application in the past.

If they did, this indicates that the new mutation op-

erators are of relevance insofar that they might have

helped prevent those errors as they would have moti-

vated improving the test suite in a way to catch such

errors and would have generally raised awareness for

these types of errors. To gain insights into the mis-

takes made in the past, both the bug tickets and the

code review ﬁndings were consulted. In the follow-

ing, for each new mutation operator, the bug tickets

and code review ﬁndings related to that new mutation

operator are brieﬂy discussed.

The mistake of using a loose comparison operator

instead of a strict one, which is simulated by the ﬁrst

new mutation operator, was uncovered in one code re-

view. The mistake simulated by the second mutation

operator, i.e., the mistake of forgetting to catch an er-

ror, was addessed in one code review ﬁnding and two

bug tickets. The third mutation operator is the muta-

tion operator artiﬁcially inserting the mistake of set-

ting a wrong default value. This mistake was found in

one bug ticket as well. The fourth mutation operator

Mutation Operators for Mutation Testing of Angular Web Applications

395

simulates the mistake of forgetting input validation.

This mistake was addressed in three bug tickets and

one code review ﬁnding. The ﬁfth and sixth new mu-

tation operator are the mutation operators for chang-

ing the name of a form ﬁeld and for forgetting to sub-

scribe to an Observable. For those two mistakes, no

occurrence could be found. This could be due to the

reason that both of these errors might become appar-

ent already when testing the implementation locally

during development. Therefore, maybe, the mistakes

are noticed too early to be found in a code review or

bug ticket. The seventh mutation operator is the one

simulating the mistake of forgetting to unsubscribe an

Observable. This mistake was found in two code

review ﬁndings. The eighth and last new mutation

operator is the one simulating the wrong use of the

RxJS operators. This mistake was also addessed in

one code review.

For the traditional mutation operators in Stryk-

erJS, it was also checked whether matching bug tick-

ets or code review ﬁndings exist. For those mutation

operators, however, only mistakes corresponding to a

removal of the sort() method, an inverted boolean

value, and several wrong string literals were found. It

thus becomes apparent that the new mutation opera-

tors are, in comparison to the traditional mutation op-

erators, more successful in simulating real-world pro-

gramming mistakes.

6.4 Developer Feedback

To evaluate the new mutation operators, a semi-

structured expert interview was conducted with the

main developer of the web application under consid-

eration. He set up the initial application, has been on

the team ever since, and is the team member doing

most code reviews.

Even though he had not come across mutation

testing in practice yet, the developer said he would

expect mutation testing to provide an additional level

of quality. According to the developer, the mistakes

occurring in Angular web applications are typically

not generic mistakes, but mostly frontend-, web- and

Angular-speciﬁc mistakes. This indicates that the tra-

ditional mutation operators inserting generic mistakes

do not represent typical mistakes in Angular web ap-

plications. This was also supported by the answers

of the developer after being shown the traditional mu-

tation operators implemented into the StrykerJS tool.

He said that most of those mistakes are unlikely to

happen in reality, especially the very basic changes

like switching an operator. The only mutation op-

erator mentioned to maybe be realistic was the one

changing the condition of, e.g., a loop or an if state-

ment. Most of the traditional mutation operators,

however, would not be realistic. Additionally, it was

said that if mistakes such as the ones inserted by the

traditional mutation operators were made, they would

be recognized rather early. Thus, the interview not

only suggests that the traditional mutation operators

do not represent realistic mistakes, but also that if

such mistakes occur, they are recognized early, there-

fore indicating a lower need for test cases catching

those types of mistakes. The new mutation operators,

in contrast, were assessed as mostly being very real-

istic by the developer.

The developer stated that he would prefer the new

mutation operators over the traditional mutation oper-

ators, indicating that the new mutation operators pose

a useful contribution to practice. He would not use

both the new and the traditional mutation operators in

combination, because of the human effort and limited

time of the developers required for checking the muta-

tion testing result and ﬁxing the test suite. If, however,

the human effort would be lower than expected, more

mutation operators could be added. Overall, the in-

terview indicates that the new mutation operators are

an improvement over the traditional mutation opera-

tors and provide beneﬁts for practitioners in the ﬁeld

of Angular web applications.

6.5 Discussion

Summarizing the results of the four previous subsec-

tions, it can be said that the eight new mutation oper-

ators successfully insert more realistic mistakes than

the traditional mutation operators. The new mutation

operators were additionally found to introduce on av-

erage fewer mutants into the code than the traditional

mutation operators. If used instead of the traditional

mutation operators, they can contribute to reducing

the execution time and possibly also the human ef-

fort. Another insight gained is that – in the case of the

web application under consideration – the new mu-

tation operators inserted more changes into the areas

of code already covered by tests, whereas the tradi-

tional mutation operators inserted more changes into

the uncovered areas. The traditional mutation oper-

ators thus motivate the developers to write more test

cases for uncovered parts of the code, while the new

mutation operators motivate the developers more to

improve the test cases which already cover the mu-

tant but do not manage to kill it. An additional in-

sight is that even though the new mutation operators

inserted a higher percentage of mutants into areas of

code already covered by tests, their use still resulted

in a lower rate of killed mutants than the traditional

mutation operators. This might indicate that the new

ENASE 2025 - 20th International Conference on Evaluation of Novel Approaches to Software Engineering

396

mutation operators insert hard-to-kill changes which

require more speciﬁc test cases.

7 CONCLUSIONS

In this work, eight new mutation operators for the

TypeScript ﬁles of Angular web applications were in-

troduced. Towards that goal, typical mistakes made

by programmers when developing Angular web ap-

plications were identiﬁed, and, based on them, new

mutation operators were designed that simulate such

mistakes. The mutation operators were implemented

to allow applying them to Angular web applications in

practice. Finally, the results were demonstrated and

evaluated to allow assessing whether the new muta-

tion operators provide additional beneﬁts compared

to the existing, traditional mutation operators.

The promising results of the evaluation show that

the new mutation operators can contribute to practice

by allowing Angular developers to better evaluate the

quality of their test suites. Practitioners can apply the

new mutation operators to their Angular web applica-

tions and can use the mutation testing results as guide-

lines to assess the current quality of the test suites and

to improve it so that both current and future mistakes

in the application code are more likely to be identi-

ﬁed. When using the new mutation operators instead

of the traditional mutation operators, they can addi-

tionally contribute to reducing the execution time and

human effort associated with mutation testing.

Threats to validity consist of the evaluation of the

mutation operators on a single web application and

by consulting only the main developer. To conﬁrm

the generalizability of the ﬁndings, as part of future

work, the new mutation operators should be tried out

on more Angular applications and more stakeholders

should be interviewed to gain their insights on the po-

tential beneﬁts and drawbacks. Similarly, a threat to

validity is that the interviews used to identifying com-

mon mistakes might be biased based on personal ex-

periences and having a similar background. As part of

future work, more interviews with developers of vari-

ous levels of experience might be conducted and more

mutation operators could be added that simulate the

newly identiﬁed common mistakes. Another possibil-

ity for future work would be to apply the approach

of this work to other domains or frameworks. All in

all, it becomes clear that the promising results of this

work open up a lot of areas of future work, illustrat-

ing the high research potential of mutation operators

simulating realistic mistakes in the ﬁeld of mutation

testing.

REFERENCES

Harman, M., Jia, Y., and Langdon, W. B. (2010). A man-

ifesto for higher order mutation testing. In 2010

Third International Conference on Software Testing,

Veriﬁcation, and Validation Workshops, pages 80–89.

IEEE.

Info Support (2023). Stryker mutator: Supported mutators.

Jansen, N. (2020). Stryker mutator: Announcing stryker 4.0

- mutation switching.

Jia, Y. and Harman, M. (2009). Higher order muta-

tion testing. Information and Software Technology,

51(10):1379–1393.

Jia, Y. and Harman, M. (2011). An analysis and survey of

the development of mutation testing. IEEE Transac-

tions on Software Engineering, 37(5):649–678.

Lesh, B., Driscoll, D., Wortmann, J.-N., Jamieson, N., Tay-

lor, P., and Lee, T. (2022). Observable.

Lesh, B., Driscoll, D., Wortmann, J.-N., Jamieson, N., Tay-

lor, P., and Lee, T. (2023). Rxjs operators.

Malcher, F., Koppenhagen, D., and Hoppe, J. (2023). An-

gular: Das große Praxisbuch - Grundlagen, fort-

geschrittene Themen und Best Practices. ix Edition.

dpunkt Verlag, Heidelberg, 4.,

uberarbeitete und aktu-

alisierte auﬂage edition.

Marchetto, A., Ricca, F., and Tonella, P. (2009). An em-

pirical validation of a web fault taxonomy and its us-

age for web testing. Journal of Web Engineering,

8(4):316–345.

Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves

Le Traon, and Mark Harman (2019). Chapter six -

mutation testing advances: An analysis and survey. In

Atif M. Memon, editor, Advances in Computers, vol-

ume 112, pages 275–378. Elsevier.

Mozilla (2023). Mdn web docs: Equality comparisons and

sameness.

Ocariza, F., Bajaj, K., Pattabiraman, K., and Mesbah, A.

(2013). An empirical study of client-side javascript

bugs. In 2013 ACM / IEEE International Sympo-

sium on Empirical Software Engineering and Mea-

surement, pages 55–64. IEEE.

Offutt, A. J. and Untch, R. H. (2001). Mutation 2000: Unit-

ing the orthogonal. In Wong, W. E., editor, Mutation

Testing for the New Century, pages 34–44. Springer

US, Boston, MA.

Petrovi

c, G. and Ivankovi

c, M. (2018). State of mutation

testing at google. In Paulisch, F. and Bosch, J., editors,

Proceedings of the 40th International Conference on

Software Engineering: Software Engineering in Prac-

tice, pages 163–171, New York, NY, USA. ACM.

anchez, A. B., Delgado-P

erez, P., Medina-Bulo, I., and Se-

gura, S. (2022). Mutation testing in the wild: ﬁndings

from github. Empirical Software Engineering, 27(6).

vom Brocke, J., Simons, A., Niehaves, B., Niehaves, B.,

Reimer, K., Plattfaut, R., and Cleven, A. (2009). Re-

constructing the giant: On the importance of rigour in

documenting the literature search process. In Newell,

S., Whitley, E., Pouloudi, N., Wareham, J., and Mathi-

assen, L., editors, Information systems in a globalising

world, volume 161, Verona.

Mutation Operators for Mutation Testing of Angular Web Applications

397