SmartMuVerf: A Mutant Veriﬁer for Smart Contracts

Sangharatna Godboley

and P. Radha Krishna

NITMINER Technologies Private Limited,

Department of Computer Science and Engineering,

National Institute of Technology Warangal, Telangana, India

Keywords:

Smart Contract, Software Testing, Mutation Veriﬁcation.

Abstract:

Smart contracts are the logical programs holding the properties in Blockchain. These Blockchain technologies

enable society towards trust-based applications. Smart contracts are prepared between the parties to hold their

deals. If the deal held by a smart contract is complex and non-trivial, then there is a high chance of attracting

issues and loss of assets. These contracts also consider expensive assets. This necessitates the veriﬁcation and

testing of a smart contract. Since we have the source code of a smart contract, then it is reasonable to apply

veriﬁcation and testing techniques. From the traditional ways, it has been observed that mutation testing is

one of the important testing techniques. But, this testing technique suffers from the issues of time and cost. It

is true that fault-based testing is a good mechanism to perform. So, looking at the issues we introduce a new

technique for Mutation Veriﬁcation for Smart Contracts. In this paper, we present an approach for measuring

the mutation score using a veriﬁcation approach. We experimented with a total of 10 smart contracts.

1 INTRODUCTION

A blockchain is a continuously expanding list of

records, known as blocks, that are securely linked

together via cryptography (Morris, 2016)(Popper,

2016). A cryptographic hash of the preceding block,

a timestamp, and transaction data are all included in

each block. This is typically depicted as a Merkel

Tree, with leafs representing data nodes. The times-

tamp veriﬁes that the transaction data was there at

the time the block was released, allowing it to be

hashed. Because each block contains information

about the one before it, they form a chain, with each

new block reinforcing the preceding ones. As a result,

blockchains are resistant to data tampering since the

data in any one block cannot be changed retrospec-

tively without affecting all subsequent blocks.

Blockchains are distributed data structures to store

the agreed sequence of transactions in a user network.

They can be employed in many applications (e.g.,

such as banking, insurance, health applications, vehi-

cle networks, shipping, logistics, and cyber-security)

that need data exchange between different users. The

actions are stored in the form of a block and the

data is distributed over individual nodes after accept-

https://orcid.org/0000-0002-6169-6334

https://orcid.org/0000-0001-8298-7571

ing by the respective user. One of the major advan-

tages of blockchain technologies is to avoid tamper-

ing with data (that is, immutability) by anonymous

users, which in turn increases transparency and secu-

rity. In addition, blockchain has several unique and

desirable properties including decentralization, audi-

bility, anonymity, and autonomously enforcing logic

via a smart contract.

Blockchains (such as Hyperledger (Buterin et al.,

2014) and Ethereum (Dannen, 2017; etherscan,

2021)) enforce consensus, if any, by the users in-

volved, as deﬁned in the smart contract. A smart con-

tract is a computer program, comprising executable

codes, residing on the blockchain and executed once

speciﬁc (pre-deﬁned) conditions are met. The trans-

actions are programmable by smart contracts. A

smart contract pays attention to transactions sent to

it, executes application logic upon receipt of a trans-

action, and depending on the need they can generate

other transactions that can be received by participat-

ing users. Thus, a smart contract includes code and

data on which the smart contract operates. Also, a

smart contract can control other smart contracts.

The code for smart contracts is typically written

in a high-level programming language such as Go

“The Go programming language,” https://golang.org/.

346

Godboley, S. and Krishna, P.

SmartMuVerf: A Mutant Veriﬁer for Smart Contracts.

DOI: 10.5220/0011822200003464

In Proceedings of the 18th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2023), pages 346-353

ISBN: 978-989-758-647-7; ISSN: 2184-4895

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

for Hyperledger

and Solidity

for Ethereum. Similar

to any other software application, there may be devi-

ations, errors and vulnerabilities in the smart contract

logic code. Smart contract-enabled blockchains guar-

antee that conditions in a smart contract are not modi-

ﬁed once they have been written and published. Thus,

coding smart contract logic as per the application re-

quirements and ensuring the correctness of that code

is very important and challenging. To the best of our

knowledge, there is very limited work in the literature

on the veriﬁcation and testing of smart contract logic.

A Smart Contract is a computer program, some-

times described as a transaction protocol, that is de-

signed to execute, control, or document legally sig-

niﬁcant events and activities in accordance with the

provisions of a contract or agreement (Martin and

Boris, 2019). The decrease in the need for trusted in-

termediaries, arbitration and enforcement costs, fraud

losses, and purposeful and inadvertent exceptions are

all goals of smart contracts. Scripting languages have

been supported by numerous cryptocurrencies since

Bitcoin, allowing for more complex smart contracts

between untrusted parties.

These Smart Contracts are programs that are col-

lectively run by a network of mutually distrusting

nodes that use a consensus protocol like proof-of-

work or proof-of-stake to digitally enforce agree-

ments between nodes (Maher and van Moorsel,

2019). The smart contract’s code cannot be modi-

ﬁed or its execution subverted by the nodes or the

smart contract’s developer. Smart contracts have been

used in a variety of industries, including ﬁnance, in-

surance, identity management, and supply chain man-

agement since they were conceived and implemented

by blockchain platforms like Ethereum and EOS. In

our project, we are using Ethereum as the smart con-

tract platform, with smart contracts written in the So-

lidity programming language.

Smart contracts are vulnerable to hacking because

they are entrusted by users with handling and transfer-

ring assets of signiﬁcant value. Because the contract

becomes immutable once deployed on the blockchain,

such hacking is more dangerous than on a traditional

network system, essentially creating a high-risk, high-

stake paradigm: the deployed code is virtually im-

possible to patch, and contracts collectively control

billions of dollars in digital assets. For example,

there have been numerous well-publicized attacks on

Ethereum smart contracts: the reentrancy attack suc-

cessfully stole $60 million in tokens from a contract,

leading to the hard fork that established Ethereum

“Hyperledger project,” https://www.hyperledger.org/.

“Solidity smart-contract language,”

https://solidity.readthedocs.io/.

Classic (ETC).

Testing of Smart Contracts is important. The

widespread acceptance of smart contracts has ce-

mented their place in the next-generation blockchain

technology ecosystem. Writing a correct smart con-

tract, on the other hand, is notoriously difﬁcult. Fur-

thermore, once a network conﬁrms a state-changing

transaction, the result is immutable. As a result, ex-

tensive testing of a smart contract application is es-

sential prior to its implementation.

Smart contracts are deployed and executed in a

blockchain environment, where the transactions are

irreversible. Smart contracts themselves are im-

mutable, which ensures their tamper-proof nature.

Hence, once the smart contract is deployed, improv-

ing the code or ﬁxing the bugs is not possible.

In this work, we propose an approach that ensures

conformance of execution as per the smart contract

logic, if any, of the execution trace of a smart contract.

To do this we propose the Mutation Veriﬁcation tech-

nique. The original smart contract serves as a master

smart contract. We create Logical Operator Replace-

ment (LOR) and Relational Operator Replacement

(ROR) mutants for Original Smart Contract (OSC).

Our approach allows annotating the smart contract

with goal constraints or targets in the form of “as-

serts" that ensure the speciﬁcation of all created mu-

tants. The solidity compiler with a bounded model

checking feature detects the targets in annotated smart

contracts. Later, we extract useful information from

the execution report and show the mutation analysis

report. The main contribution of this work is to pro-

vide a platform for the contributors of smart contracts

to test their scripts.

The rest of the paper is organized as follows. In

Section 2, we discuss the related work. Section 3

shows basic concepts. Section 4 presents the pro-

posed approach. Section 5 describes the experimental

results. Finally, We conclude the paper with future

insights in Section 6.

2 RELATED WORK

The ﬁrst blockchain platform that supports smart con-

tracts is Ethereum (Dannen, 2017), in which Solidity

scripting language is used for developing smart con-

tracts. There are few more blockchain platforms such

as Hyperledger Fabric (Androulaki et al., 2018), Bit-

coin (Böhme et al., 2015), and RootStock (Reighard

et al., 2008), that supports the deployment and execu-

tion of smart contracts. There are some work such as

(Sánchez-Gómez et al., 2019; Liu et al., 2020a) in the

domain of smart contract testing.

SmartMuVerf: A Mutant Veriﬁer for Smart Contracts

347

(Andesta et al., 2020) have proposed a method of

testing solidity smart contract using mutation testing

and proposed 10 classes of mutation operators. Their

proposed method is capable of regenerating the real

bug in ten contracts out of ﬁfteen contracts and they

have also provided their mutation operators with uni-

versal mutator tool. (Peng et al., 2019) suggested a

full system for analyzing Solidity contracts, querying

code creation, and instrumentation. It helps Solidity

contract developers and testers create source-level ap-

proaches for analysis, code generation, diagnostics,

and optimizations.

(Driessen et al., 2021) introduced a method that

generates an automatic test suite for stand-alone con-

tracts using two search algorithms. They tested a col-

lection of 36 real-world Contracts to evaluate both

the algorithms, a genetic algorithm, and a random

search technique. VerX by (Permenev et al., 2020)

is a temporal property veriﬁcation framework. To as-

sess these features, they use symbolic execution and

abstract interpretation-based predicate abstraction, as

well as their own policy language. The need of exten-

sive testing of smart contracts before implementation

is demonstrated by Xinming Wang et al. (Xinming

et al., 2019). They proposed the concepts of full trans-

action basis path set and bounded transaction interac-

tions to represent the basic control ﬂow behaviours of

smart contracts. They developed a set of path-based

test coverage criteria based on these two ideas. They

also conducted a case study to evaluate the effective-

ness of the proposed test coverage criteria to random

testing and statement coverage testing, ﬁnding that k-

bounded transaction coverage testing is nearly 55 per-

cent more effective than statement coverage testing in

detecting ﬂaws (Xinming et al., 2019). They ﬁnished

by recognising the paucity of research on how to test

smart contract applications systematically.

Existing approaches could not efﬁciently perform

mutation testing for integer overﬂow in Ethereum

Smart Contracts (ESCs), according to Jinlei Sun, et

al. (Jinlei et al., 2020). As a result, they presented

ﬁve speciﬁc mutation operators to address those ﬂaws

in ESC testing sufﬁciency detection. The different

integer overﬂow problems can be reliably simulated

using these operators. Their empirical study on 40

open-source ESCs to evaluate the effectiveness of the

proposed mutation operators revealed that the pro-

posed mutation operators reproduced all 179 inte-

ger overﬂow vulnerabilities in the 40 smart contracts,

and the generated mutants had a high compilation

pass rate and integer overﬂow vulnerability genera-

tion rate. Furthermore, the generating mutants dis-

covered the ﬂaws in existing testing methodologies

for integer overﬂow vulnerabilities.

Xingya Wang, et al. (Xingya et al., 2019) also

emphasise the signiﬁcance of doing adequate testing

of Ethereum Smart Contracts (ESC), and present a

multi-objective random and NSGA-II based approach

to ﬁnd cost-effective test-suites. Their approach is

the ﬁrst Pareto minimisation approach to ESC testing,

combining the goal of reducing uncovered branches

in conventional software with the goal of reducing the

time and gas cost of testing ESCs. Their empirical as-

sessment of a collection of smart contracts in eight

of the most popular Ethereum Decentralised Appli-

cations also conﬁrmed that the proposed ways could

drastically cut gas and time costs while maintaining

branch coverage.

Gustavo Greico, et al. (Gustavo et al., 2020) intro-

duce Echidna, an open-source smart contract fuzzer

that automates the generation of tests to discover as-

sertion and custom property violations. Echidna’s

major beneﬁt is that it doesn’t require any compli-

cated conﬁguration or contract deployment to a local

blockchain. It has been used in over ten large-scale

paid security audits, with comments from those au-

dits helping to enhance the usability and test gener-

ation methodologies. They state that recent research

examining and categorising ﬂaws in critical contracts

found that fuzzing with custom user-deﬁned proper-

ties can detect up to 63 percent of the most severe

and exploitable ﬂaws in contracts, implying that smart

contract developers and security auditors have a sig-

niﬁcant need for high-quality, easy-to-use fuzzing.

Echidna supports three properties: User-deﬁned, as-

sertion veriﬁcation, and gas use estimation.

Smart contracts are vulnerable to hacking, ac-

cording to Jean-Wei Liao et al. (Jian-Wei et al.,

2019), because they are difﬁcult to ﬁx and lack assess-

ment standards to ensure their quality. They present

SoliAudit, a smart contract vulnerability assessment

tool that employs machine learning and fuzz test-

ing. They developed a Gray-box fuzz testing mech-

anism that includes a fuzzer contract and a simulated

blockchain environment for online transaction veriﬁ-

cation. The accuracy of SoliAudit can approach 90

percent, and the fuzzing can assist detect possible

weaknesses, such as reentrancy and arithmetic over-

ﬂow concerns, according to their real-world evalu-

ation utilising nearly 18k smart contracts from the

Ethereum blockchain and CTF samples.

Erfan Andesta et al. (Erfan et al., 2020) suggested

a mutation-based testing technique for smart contracts

in the Solidity language. They looked at a long list of

known defects in Solidity smart contracts and came

up with ten different mutation operators based on the

actual ﬂaws. Classic Mutation Operators and Solidity

Mutation Operators were used (as the Classic Muta-

ENASE 2023 - 18th International Conference on Evaluation of Novel Approaches to Software Engineering

348

tion Operators were not enough). Their tests revealed

that their operators can regenerate real defects for 10

out of 15 well-known problematic smart contracts.

Model-based Software development, and mod-

elling techniques, such as use cases models and ac-

tivity diagram models based on Uniﬁed Model Lan-

guages, are combined by N. Sánchez-Gómez et al.

(Sanchez et al., 2019) to simplify and improve the

modelling, management, and execution of collabora-

tive business processes between multiple companies

in the Blockchain network. Current testing and anal-

ysis methods, according to Ye Liu et al. (Liu et al.,

2020b), lack support for contracts that are frequently

larger and more complicated. They demonstrate Mod-

Con, a model-based testing platform that uses user-

deﬁned models to design test oracles, drive test gen-

eration, and assess test adequacy. ModCon was able

to reach all states and transitions for each scenario in

roughly 500 test cases in their experiments.

Purathani Praitheeshan, et al. (Purathani et al.,

2019) investigated 16 smart contract vulnerabilities

and discovered that some of them need adequate solu-

tions. They projected that numerous attacks are yet to

be exploited by associating 16 Ethereum vulnerabili-

ties with 19 software security concerns. They discuss

smart contract security issues, as well as the existing

analytical tools and detection approaches. They iden-

tify and explain the primary weaknesses in smart con-

tracts that could cause serious issues.

3 BASIC CONCEPTS

In this section, we discuss some important terminolo-

gies used in this paper.

Deﬁnition 3.1 (Smart Contract). As per IBM

“Smart

contracts are scripts stored on a blockchain that run

when all the previously essential requirements are

full-ﬁlled. Smart contracts are used to automate the

process of an agreement so that all contributors can

be immediately certain of the outcome, without any

intermediary’s involvement or time loss. These can

be automated in a workﬂow, triggering the next ac-

tion when requirements are met."

Deﬁnition 3.2 (Bounded Model Checking (BMC)).

“In Bounded Model Checking (BMC), a Boolean for-

mula is constructed which is satisﬁable if and only

if the underlying state transition system can realize a

ﬁnite sequence of state transitions that reach certain

states of interest. If such a path segment cannot be

found at a given length, k, the search is continued for

larger k. The procedure is symbolic, i.e., symbolic

https://www.ibm.com/topics/smart-contracts

Boolean variables are utilized; thus, when a check

is done for a speciﬁc path segment of length k, all

path segments of length k are being examined. The

Boolean formula that is formed is given to a satisﬁa-

bility solving program and if a satisfying assignment

is found, that assignment is a witness for the path seg-

ment of interest" by (Clarke et al., 2004).

Deﬁnition 3.3 (Mutation Veriﬁcation). “A fault-

based veriﬁcation approach where the mutants are tar-

geted as goal constraints and the veriﬁcation approach

proves the targets with counter-examples as to show

the killed mutants."

4 PROPOSED APPROACH

In this section, we discuss the framework of the

proposed approach with an algorithmic description.

Also, we provide a detailed explanation with a work-

ing example.

4.1 Framework of SmartMuVerf

Fig.1 shows a schematic representation of our pro-

posed approach SmartMuVerf. This framework

mainly contains four components 1. Mutator 2. Mu-

tants Annotator, 3. SolBMC, and 4. Extractor. The

ﬂow starts with supplying Original Smart Contract

into Mutator to create LOR and ROR types mu-

tants. Now, these created mutants along with Orig-

inal Smart Contract supplied into Mutants Annotator

to produce Annotated Smart Contract. This modiﬁed

version of the smart contract has a goal constraint for

each mutant. These goal constraints have been de-

signed and injected in the form of assertions.

Next, the Annotated Smart Contract is supplied

into a smart veriﬁer SolBMC. Since SolBMC follows

SMT technique using Z3 constraints solver, the reach-

ability and feasibility of each marked assertion can be

done. The SolBMC generates a detailed execution re-

port. This execution report contains the log of each

assertion violation with the counter-example (test in-

puts). The Extractor component analyses the Execu-

tion Report and identiﬁes the total number of asser-

tion violations. Each assertion violation represents a

killed mutant. Finally, Killed and Alive mutants re-

ports are generated from Extractor.

4.2 Algorithmic Description

In this section, we explain our proposed approach

with algorithmic descriptions.

Algorithm 1 shows the implementation of the

SmartMuVerf. The main input to this algorithm is

SmartMuVerf: A Mutant Veriﬁer for Smart Contracts

349

Original Smart

Contract

Mutator

Mutants

SmartMuVerf

Mutants

Annotator

Annotated Smart

Contract

SOLBMC

Execution

Report

Extractor

Killed

Mutants

Alive

Mutants

Figure 1: The framework for the proposed approach Smart-

MuVerf.

Original Smart Contract (OSC) to produce Mutation

Score Percentage (MScore%). Line 3 of Algorithm 1

invokes Algorithm 2 which shows the implementation

of Mutator component. Line 4 of Algorithm 1 invokes

Algorithm 5 which shows the implementation of Mu-

tants Annotator to produce Annotated Smart Contract

(ASC) as output. Line 5 of Algorithm 1 shows the

processing of Solidity Compiler with Bounded Model

Checker (SolBMC) by supplying ASC and generation

Execution_Report. Line 6 of Algorithm 1 shows the

processing of Extractor component. It requires the

Execution_Report to identify Killed and Alive mu-

tants. Line 7 of Algorithm 1 shows the formula to

compute MScore% using Killed Mutants and Total

Mutants.

Next, let’s see the detail of Algorithm 2 named

Mutator which takes Original Smart Contract (OSC)

and produces mutants viz. {M1,M2,M3..Mn}. Line

3 of Algorithm 2 invokes Algorithm 3 RM (ROR

Mutator). Similarly, Line 4 of Algorithm 2 invokes

Algorithm 3 LM (LOR Mutator). Line 5 of Algo-

rithm 2 renames all the collected mutants so that

they can be identiﬁed uniquely and annotated in a

smart contract. Algorithm 3 takes OSC and produces

{RM1, RM2, RM3..RMn}. Line 3 iterates the loop for

each condition in the program and extract the rela-

tional operator so that mutants can be created. Lines

6 to 23 show that for each relational operator rest 5

operators create the mutants. Finally, Line 25 return

all the create ROR mutants {RM1,RM2,RM3..RMn}.

Similarly, Algorithm 4 shows the implementation of

LM (LOR Mutator). It takes OSC as input and pro-

duces LOR mutants {LM1,LM2,LM3,...,LMn}.

Algorithm 5 shows the implementation of the Mu-

tator_Annotator. Here we supply Original Smart

Contract OSC as an input and get Annotated Smart

Contract ASC as an output. Line 3 of Algorithm

5 shows an iteration for all the predicates identiﬁed

from OSC. The loop iterates for each predicate iden-

tiﬁed at a speciﬁc line. For each predicate, we iter-

ate all the created mutants speciﬁc to that predicate

only to create the goal constraints. Our template is

“assert(!((Pred

)!=(M

)));" for each mutant. We in-

ject this goal constraint just above the Pred

, where

i shows the current predate and j shows the current

mutant. The aim of this template is to show whether

the considered mutant is Killed or Alive after execu-

tion. If SolBMC shows the assertion violation hap-

pens then the mutant is considered killed otherwise

alive.

Algorithm 1: SmartMuVerf.

1: Input: {OSC}

2: Output: {MScore%}

3: {M1, M2, M3..Mn} ← Mutator(OSC)

4: ASC ← Mutants_Annotator(OSC, {M1, M2, M3,

....Mn})

5: Execution_Report ← SolBMC(ASC)

6: {KilledMutants, AliveMutants} ←Extractor(Ex

-ecution_Report)

7: MScore%← Div(Killed Mutants,Total Mutants)

X 100

Algorithm 2: Mutator.

1: Input: {OSC}

2: Output: {M1,M2,M3..Mn}

3: {RM1, RM2, RM3..RMn} ← RM(OSC)

4: {LM1, LM2, LM3..LMn} ← LM(OSC)

5: {M1, M2, M3..Mn} ←Rename({RM1,RM2,RM3,

...,RMn}+{LM1,LM2,LM3,...,LMn})

4.3 Working Example

In this section, we consider an example smart contract

namely RoomThermostat.sol from the set of 10 smart

contracts. The original smart contract for RoomTher-

mostat.sol is shown in Listing 1(Artifacts:, 2022)

The characteristics for RoomThermostat.sol wrt. size

is 51 LOCs, 3 functions, 3 predicates, and 6 atomic

conditions. In a smart contract the predicate and con-

ditions can be written in if-else, if-else-if, for-loop,

while-loop and require statements. We only consider

two classes of faults viz. LOR (Logical Operator Re-

placement) and ROR (Relational Operator Replace-

ment). In LOR, we replace ∥ with && and vice-

versa. Similarly in ROR, we take one relational oper-

To save space in the paper we have uploaded all the

listings (Listings 1 to 8) at (Artifacts:, 2022).

ENASE 2023 - 18th International Conference on Evaluation of Novel Approaches to Software Engineering

350

Algorithm 3: RM (ROR Mutator).

1: Input: OSC

2: Output: {RM1,RM2,RM3,RM4,...,RMn}

3: while Condition ∈ AllConditions do

4: OP ← OperatorExtractor(Condition)

5: Update Mutant ids

6: if OP == ‘<’ then

7: RM1 = ‘! =’, RM2 = ’>’, RM3 = ’<=’,

RM4 = ’>=’, RM5 = ’==’

8: end if

9: if OP == ‘>’ then

10: RM1 = ‘! =’, RM2 = ’<’, RM3 = ’<=’,

RM4 = ’>=’, RM5 = ’==’

11: end if

12: if OP == ‘<=’ then

13: RM1 = ‘! =’, RM2 = ’<’, RM3 = ’>’,

RM4 = ’>=’, RM5 = ’==’

14: end if

15: if OP == ‘>=’ then

16: RM1 = ‘! =’, RM2 = ’<’, RM3 = ’<=’,

RM4 = ’>’, RM5 = ’==’

17: end if

18: if OP == ‘==’ then

19: RM1 = ‘! =’, RM2 = ’<’, RM3 = ’>’,

RM4 = ’<=’, RM5 = ’>=’

20: end if

21: if OP == ‘! =’ then

22: RM1 = ‘==’, RM2 = ’<’, RM3 = ’>’,

RM4 = ’<=’, RM5 = ’>=’

23: end if

24: end while

25: return {RM1,RM2,RM3,RM4,...,RMn}

ator from six operators (>,>=,<,<=,==,!=) and replace

it with other ﬁve relational operators.

Listing 2(Artifacts:, 2022) shows the total LOR

and ROR mutants created for smart contracts in List-

ing 1(Artifacts:, 2022). There are three predicates

and six atomic conditions at lines 26, 36, and 45 of

Listing 1(Artifacts:, 2022). The mutants shown at

lines 1 to 11 in Listing 2(Artifacts:, 2022) are for line

26 i.e. “if (Installer != msg.sender ∥ State != State-

Type.Created)” in Listing 1(Artifacts:, 2022). The

mutants shown at lines 12 to 22 in Listing 2(Artifacts:,

2022) are for the line 36 i.e. “if (User != msg.sender

∥ State != StateType.InUse)” in Listing 1(Artifacts:,

2022). Similarly, the mutants shown at lines 13 to 23

in Listing 2(Artifacts:, 2022) are for the line 35 i.e.

“if (User != msg.sender ∥ State != StateType.InUse)”

in Listing 1(Artifacts:, 2022).

Next, using Mutants Annotator component of

our proposed approach, we generate the Annotated

Smart Contract version i.e RoomThermostat_mod.sol

as shown in Listing 3(Artifacts:, 2022). In this ver-

Algorithm 4: LM (LOR Mutator).

1: Input: OSC

2: Output: {LM1,LM2,LM3,LM4,...,LMn}

3: while Predicates ∈ AllPredicates do

4: OP ← OperatorExtractor(Predicates)

5: Update Mutant ids

6: if OP == ‘&&’ then

7: LM1 = ‘∥’

8: end if

9: if OP == ‘∥’ then

10: LM1 = ‘&&’

11: end if

12: end while

13: return {LM1,LM2,LM3,LM4,...,LMn}

Algorithm 5: Mutants Annotator.

1: Input: OSC

2: Output: ASC

3: while Pred

∈ OSC do

4: while M

∈ {M1,M2,M3,M4,...,Mn} do

5: Create and In-

ject “assert(!((Pred

)!=(M

)));" above the

Pred

6: end while

7: end while

8: return ASC

sion, we inject the targets or goal constraints using the

“assert” syntax just above the predicates or conditions

identiﬁed in the contract. We prepare the target for

example “assert(!((Installer != msg.sender ∥ State !=

StateType.Created) != (Installer !=msg.sender &&

State != StateType.Created)));” for the LOR mu-

tant “Installer != msg.sender && State != State-

Type.Created)” which is from line 26 i.e. “if (Installer

!= msg.sender ∥ State != StateType.Created)” in List-

ing 1(Artifacts:, 2022). Note that the assertion func-

tion is used to identify whether the mutant is killed

or alive. If assertion violation happens for the target

asserted then we claim that the mutant is killed, other-

wise alive. Deeper to this, please see the “!=” operator

between the ﬁrst expression and the second expres-

sion. The ﬁrst expression in the assert is the original

code whereas the second expression in the assert is

the mutated code. Now we use traditional deﬁnitions

of killed mutants and alive mutants. As we know, if

the outputs of the original code and mutated code are

different then the mutant is considered as killed other-

wise alive. Here we compare the context of the state

until the point of the reached line where the mutation

is applied. We consider that the line can be reached

via. one or more different path(s). So our objective

in using “!=” is to show the killing of a mutant. Be-

SmartMuVerf: A Mutant Veriﬁer for Smart Contracts

351

cause, if this non-equality (different outputs or val-

ues) is true then the assertion will be violated. Note

that only a single path is sufﬁcient to show the killing

of a mutant. Now, consider this non-equality which is

false for all the paths reaching to that line and never

shows the assertion violation. It means there exists no

feasible path that a mutant cannot be killed, hence it is

an alive mutant. Finally, taking all the mutants from

Listing 2(Artifacts:, 2022) and properly annotate as

the goal constraints to produce Annotated Smart Con-

tract as shown in Listing 3(Artifacts:, 2022).

Now, the Annotated Smart Contract as shown in

Listing 3(Artifacts:, 2022) is supplied into SolBMC to

produce the Execution Report with counter-examples

Listing 8(Artifacts:, 2022). This report has all the exe-

cution logs with the warning and messages. This exe-

cution report generated is supplied into the Extractor

component to get the Killed and alive mutants infor-

mation. As we have injected 33 targets, SolBMC has

detected 30 of them with counterexamples. So for this

contract, we have a 90.91% mutation score. This in-

formation can be found in the Killed Mutants report

in Listing 4(Artifacts:, 2022). The detailed mutation

analysis report is shown in Listing 5(Artifacts:, 2022).

In our analysis, we have also captured the exe-

cution time of testing the smart contract. The time

analysis for the working analysis is shown in List-

ing 6(Artifacts:, 2022). We can observe that SolBMC

took 1.36 sec for this example. Also, the execution

report in Listing 5(Artifacts:, 2022) contains all the

counter-examples or test inputs for each killed mutant

(assertion violation). These test cases are very much

useful.

Also, our technique can be considered as the

guided approach to generate the optimal test inputs

subject to the types of faults considered. If we had

the same set of test cases and we suppose to use the

traditional mutation testing then we needed to replay

all these unique 33 test inputs over 33 different mu-

tant versions. We can estimate or predict for sure it

cannot be ﬁnished in 1.59 sec. But, as we can see us-

ing SmartMuVerf we got the results in 1.59 sec. This

enables the usability of our approach.

5 EXPERIMENTAL RESULTS

In this section, we discuss the setup and benchmarks

tested, and discuss on results. We used an Intel®

Core™ i7-9700 CPU @ 3.00GHz Linux box (64-bit

Ubuntu 20.04.2 LTS) with 8 GB RAM and llvmpipe

(LLVM 11.0.0, 256 bits) graphics in Oracle Virtuali-

sation. We have used PPAs for Ubuntu with the latest

Table 1: Result Analysis Note: #L: Lines of Code, #ML:

Modiﬁed Lines of Code, #M: Total Mutants, #K: Total

Killed Mutants, #A: Total Alive Mutants, MS%: Mutation

Score, T: Execution Time (Sec).

Contracts #L #ML #M #K #A MS% T

BasicProvenance 53 79 27 24 3 88.89 1.21

RoomThermostat 51 84 33 30 3 90.91 1.59

SimpleMarketplace 73 101 30 29 1 96.67 1.16

Token 43 75 31 31 0 100.00 1.93

escrow 78 99 21 21 0 100.00 1.53

DogeMojo 53 85 31 31 0 100.00 2.29

ShibaAstronaut 53 85 31 31 0 100.00 2.14

UniswapV3MigratorProxy 23 29 5 5 0 100.00 0.85

payments 53 73 20 19 1 95.00 1.31

kia_quiz 51 77 26 23 3 88.46 1.25

Table 2: Aggregated Result Analysis.

#L #ML #M #K #A Avg_MScore(%) T

Total 531 787 255 244 11 95.99 15.25

stable version of Solidity Compiler

. We have used

the following command setting as shown in Listing

7(Artifacts:, 2022):

Table 1 shows the detailed results for 10 smart

contracts taken from (etherscan, 2021). #L and #ML

present the size of the contract before and after the

annotations for targets or goal constraints for created

mutants respectively. Here, #L shows the Lines of

Code and #ML shows Modiﬁed Lines of Code. Note

that the targets or goal constraints are injected into

contracts, in a way that the semantics of the contract

will not be affected. #M shows the total mutants cre-

ated for the contract. #K shows the Killed mutants.

The #A shows the Alive mutants which have been

computed using #A = #M − #K. MS% shows the mu-

tation score calculated using MS% =

. Lastly, T

shows the total execution time in seconds.

Table 2 shows the aggregated results for 10 smart

contracts. In total, we processed 531 and 787 Lines

of code for original and modiﬁed smart contracts re-

spectively. We created a total of 255 mutants and

injected them in proper locations and executed them

with SolBMC. The veriﬁer proved a total of 244 killed

mutants and 11 as Alive mutants. On an average of

10 smart contracts, we achieved 95.99% in 15.25 sec-

onds total time.

6 CONCLUSION

The main objective of this work is to verify the cre-

ated mutants for the smart contract. It is very impor-

tant to test smart contracts by looking at the critical

business in the blockchain. If an incorrect contract or

bug in the contract exists, then there is a high chance

https://docs.soliditylang.org/en/develop/installing-

solidity.html#linux-packages

ENASE 2023 - 18th International Conference on Evaluation of Novel Approaches to Software Engineering

352

of losing the expensive assets. In this paper, we pro-

posed a novel approach to computing mutation scores

for a smart contract using a solidity compiler with

a bounded model checker. We propose to use mu-

tation veriﬁcation in the industry which replaces the

traditional mutation testing methodology. Our ongo-

ing work focuses on a detailed analysis of more types

of faults. We will explore other techniques such as

Fuzzing and Symbolic execution for a more detailed

analysis.

ACKNOWLEDGEMENT

We would like to thank the Department of Science

and Technology (DST), Government of India, NM-

ICPS, IBITF IIT Bhilai for sponsoring the project to

NITMINER Technologies Private Limited (a startup

recognised by GOI) under PRAYAS scheme.

REFERENCES

Andesta, E., Faghih, F., and Fooladgar, M. (2020). Testing

smart contracts gets smarter. In 2020 10th Interna-

tional Conference on Computer and Knowledge Engi-

neering (ICCKE), pages 405–412. IEEE.

Androulaki, E., Barger, A., Bortnikov, V., Cachin, C.,

Christidis, K., De Caro, A., Enyeart, D., Ferris, C.,

Laventman, G., Manevich, Y., et al. (2018). Hyper-

ledger fabric: a distributed operating system for per-

missioned blockchains. In Proceedings of the thir-

teenth EuroSys conference, pages 1–15.

Artifacts: (2022). Raw experimnetal data. https://ﬁgshare.

com/s/9c0f08be4fbc234e8d66.

Böhme, R., Christin, N., Edelman, B., and Moore, T.

(2015). Bitcoin: Economics, technology, and gover-

nance. Journal of economic Perspectives, 29(2):213–

38.

Buterin, V. et al. (2014). A next-generation smart contract

and decentralized application platform. white paper,

3(37).

Clarke, E., Kroening, D., and Lerda, F. (2004). A tool for

checking ANSI-C programs. In TACAS, pages 168–

176. Springer.

Dannen, C. (2017). Introducing Ethereum and solidity, vol-

ume 318. Springer.

Driessen, S., Nucci, D. D., Monsieur, G., and van den

Heuvel, W.-J. (2021). Automated test-case generation

for solidity smart contracts: the agsolt approach and

its evaluation.

Erfan, A., Fthiyeh, F., and Mahdi, F. (2020). Testing smart

contracts gets smarter. 10th International Confer-

ence on Computer and Knowledge Engineering (IC-

CKE2020).

etherscan (2021). etherscan. https://etherscan.io/.

Gustavo, G., Will, S., Artur, C., Josselin, F., and Alex, G.

(2020). Echidna: Effective, usable, and fast fuzzing

for smart contracts. ISSTA ’20, July 18–22, 2020, Vir-

tual Event, USA.

Jian-Wei, L., Tsung-Ta, T., and Chia-Kang, H. (2019). So-

liaudit: Smart contract vulnerability assessment based

on machine learning and fuzz testing. 2019 Sixth In-

ternational Conference on Internet of Things: Sys-

tems, Management and Security (IOTSMS).

Jinlei, S., Song, H., Changyou, Z., and Tingyong, W.

(2020). Mutation testing for integer overﬂow in

ethereum smart contracts.

Liu, Y., Li, Y., Lin, S.-W., and Yan, Q. (2020a). Modcon:

a model-based testing platform for smart contracts. In

Proceedings of the 28th ACM Joint Meeting on Eu-

ropean Software Engineering Conference and Sym-

posium on the Foundations of Software Engineering,

pages 1601–1605.

Liu, Y., Lin, S., and Qiang, Y. (2020b). Modcon: A model-

based testing platform for smart contracts. ESEC/FSE

’20, November 8–13, 2020, Virtual Event, USA.

Maher, A. and van Moorsel, A. (2019). Blockchain-based

smart contracts: A systematic mapping study.

Martin, F. and Boris, P. (2019). Smart contracts.

Morris, D. Z. (2016). Leaderless, blockchain-based venture

capital fund raises $100 million, and counting. For-

tune.

Peng, C., Akca, S., and Rajan, A. (2019). Sif: A framework

for solidity code instrumentation and analysis.

Permenev, A., Dimitrov, D., Tsankov, P., Drachsler-Cohen,

D., and Vechev, M. (2020). Verx: Safety veriﬁcation

of smart contracts. In 2020 IEEE Symposium on Se-

curity and Privacy (SP), pages 1661–1677. IEEE.

Popper, N. (2016). A venture fund with plenty of virtual

capital, but no capitalist. NYT.

Purathani, P., Lei, P., and Jiangshan, Y. (2019). Security

analysis methods on ethereum smart contract vulnera-

bilities — a survey.

Reighard, G. L., Loreti, F., et al. (2008). Rootstock develop-

ment. The peach: Botany, production and uses. CABI

Publishing, Wallingford, Oxon, UK, pages 193–220.

Sanchez, G., Morales, T., and Torres, V. (2019). Towards

an approach for applying early testing to smart con-

tracts. 15th International Conference on Web Infor-

mation Systems and Technologies (WEBIST 2019).

Sánchez-Gómez, N., Morales-Trujillo, L., and Valderrama,

J. T. (2019). Towards an approach for applying early

testing to smart contracts. In WEBIST, pages 445–453.

Xingya, W., Haoran, W., Weisong, S., and Yuan, Z.

(2019). Towards generating cost-effective test-suite

for ethereum smart contract.

Xinming, W., Zhijian, X., Jiahao, H., and Ruihua, N.

(2019). Basis path coverage criteria for smart con-

tract application testing. 2019 International Confer-

ence on Cyber-Enabled Distributed Computing and

Knowledge Discovery.

SmartMuVerf: A Mutant Veriﬁer for Smart Contracts

353