A Comprehensive Review of Testing Blockchain Oriented Software

Mariam Lahami

, Afef Jmal Maalej

, Moez Krichen

1,2

and Mohamed Amin Hammami

ReDCAD Laboratory, National Engineering School of Sfax, University of Sfax, Tunisia

Faculty of CSIT, Al-Baha University, Saudi Arabia

Keywords:

Blockchain, Review, Dynamic Testing, Smart Contracts, BoS.

Abstract:

This work presents a short review on the most relevant studies in the context of testing Blockchain-Oriented

Software (BoS), especially at the smart contract level. Focusing on dynamic testing, we ﬁrst provide a clas-

siﬁcation of 20 studies according to the accessibility of smart contract code. Second, we give an overview

of each identiﬁed work while highlighting its advantages and limitations. Third, we discuss challenges and

opportunities in this research area. After carrying out this review, it was noticed that there is much work to be

done, especially in the context of model-based testing of smart contracts which can be a good research line for

detecting defects and enhancing the quality of such applications.

1 INTRODUCTION

Nowadays, one of the most popular trends of In-

formation Technology (IT) is the blockchain (Jab-

bar et al., 2022). Firstly introduced in the white pa-

per of Nakamoto et al. in 2008 (Nakamoto et al.,

2008), Blockchain (BC) gathered a lot of interest in

the last decade by several organizations, especially

ﬁnancial ones. BC is a distributed ledger that fol-

lows the Peer-to-Peer architecture in which each node

maintains a copy of the blockchain ledger. Its struc-

ture makes it distinguishable since it is composed of

blocks, linked together by means of cryptographic

functions, in which data are securely stored and can-

not be changed or tampered. All the participant nodes

in the blockchain network are running with respect

to a common consensus algorithm without the need

of a third party, such as ﬁnancial institutions. Cur-

rently, blockchain platforms, such as Ethereum

and

Hyperledger Fabric

, give the possibility of devel-

oping decentralized applications with complex busi-

ness logic. They introduce the concept of “smart

contracts” which is deﬁned as a collection of pro-

gramming code and data that runs on the blockchain

and it is executed to fulﬁll a given task when some

events and rules happened. By using smart contracts,

blockchain can be used not only to exchange cryp-

tocurrencies but also can be applied on several in-

dustry sectors such as property management, assur-

https://ethereum.org/en/

https://www.hyperledger.org/

ance and health (Fekih and Lahami, 2020). Although

the blockchain technology presents a novel approach

to develop decentralized applications between several

stakeholders, this technology may suffer from several

vulnerabilities and bugs within smart contracts. Due

to the decentralized nature of blockchain involving

anonymous nodes, malicious attackers may extract

digital assets from a contract, cause damage by lead-

ing a smart contract into a deadlock, which prevents

account owners from withdrawing or spending their

assets (Praitheeshan et al., 2019). For instance, 50

million dollars were stolen in the well-known ”DAO

attack”, which used a combination of smart-contract

vulnerabilities (Finley, 2016). Therefore, applying

testing techniques, before the deployment of smart

contracts, becomes a crucial task with the aim of in-

creasing conﬁdence in blockchain oriented software

and avoiding attacks and asset losses.

In this direction, several efforts were made to

detect such vulnerabilities in smart contracts. Re-

searchers have used different approaches including

theorem proving, model checking (Nelaturu et al.,

2020) and software testing (Antonino and Roscoe,

2020). Although testing blockchain oriented software

(BoS) has gained the attention of many researchers,

we have not found either recent surveys or systematic

mapping studies that provide in depth an overview

of the considered research ﬁeld, and identify exhaus-

tively the amount and quality of available research re-

sults related to it. To cope with this lack, this system-

atic literature review was conducted in which 20 stud-

Lahami, M., Maalej, A., Krichen, M. and Hammami, M.

A Comprehensive Review of Testing Blockchain Oriented Software.

DOI: 10.5220/0011042800003176

In Proceedings of the 17th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2022), pages 355-362

ISBN: 978-989-758-568-5; ISSN: 2184-4895

355

ies dealing with dynamic testing of Blockchain ori-

ented Software (BoS), dated from 2017 to November

2021, were deeply discussed. Moreover, we identi-

ﬁed several challenges and future research guidelines

related to this topic.

The organization of this survey is as follows:

Section 2 provides background information about

blockchain technology, smart contracts, and the soft-

ware testing techniques. In Section 3, related surveys

are discussed. The state-of-the-art testing approaches

and techniques for performing different types of test-

ing on smart contracts are provided in detail in Sec-

tion 4. Section 5 includes a discussion based on

blockchain testing challenges and some future direc-

tions. Finally, section 6 concludes the paper.

2 PRELIMINARIES

2.1 Blockchain

The concept of Blockchain appeared for the ﬁrst time

in the white paper of Nakamoto et al. (Nakamoto

et al., 2008) as the technology underlying Bitcoin.

Now, it garnered a lot of attention and is used in sev-

eral platforms such as Ethereum and Hyperledger. It

can be deﬁned as a distributed ledger maintained over

a peer-to-peer network.

As illustrated in Fig. 1, BC is composed of a

growing list of blocks. Each block contains essen-

tially a given number of transactions that have oc-

curred within the network. The transaction can be

seen as token transfers or any manner of data ex-

change. Each block is composed of two parts: the

header and the body. Within the body of the block,

transactions are stored while the header contains sev-

eral ﬁelds, particularly a timestamp of when the block

was produced and the identiﬁer of the previous block.

The latter is obtained by executing a cryptographic

hash function (e.g., SHA256, KECCAK256, etc.). In

this way, blocks are connected to each other like a

linked list (Ali et al., 2018).

Block 1

Header

Block 1 Body

Transactions are stored here

Hash of the previous

block header

Block 2

Header

Block 2 Body

Transactions are stored here

Hash of the previous

block header

Block 3

Header

Block 3 Body

Transactions are stored here

Hash of the previous

block header

Figure 1: Representation of a blockchain general scheme

(Ali et al., 2018).

2.2 Smart Contracts

One of the interesting features of recent blockchain

platforms like Ethereum and Hyperledger is the possi-

bility of attaching business logic code to transactions,

called Smart Contracts (SC). A SC is seen as a piece

of code stored in the blockchain, that is capable of self

execution when some conditions are met, without the

need of centralized authorizing entities.

For the case of the Ethereum platform, smart con-

tracts are written in a Turing complete language

called Solidity

and then they are compiled to the

Ethereum Virtual Machine (EVM) bytecode. Like

JavaScript language, Solidity supports features like li-

braries, inheritance and user-deﬁned types. The most

relevant feature within smart contracts is their im-

mutability. Once deployed on the blockchain, they

cannot be altered or changed. Therefore, it is highly

required to ensure their correctness and security be-

fore their deployment on the blockchain platform.

2.3 Blockchain Testing Techniques

One of the most important activities for proper soft-

ware development is testing activity. In fact, it is de-

ﬁned in (Freedman, 1991) as the process of validat-

ing and ensuring the quality of a System Under Test

(SUT). It is usually performed with the aim of assess-

ing the compliance of a system to its intended speci-

ﬁcations. Furthermore, software testing can be static

or dynamic. Static testing does not involve software

execution, but examines or analyses the source code

structure, syntax and data-ﬂow, and is usually called

Static analysis

In contrast with static testing, dynamic testing in-

volves the software execution while feeding inputs

and producing outputs. Test cases are typically devel-

oped by specifying test inputs and expected outputs.

The purpose of testing here is to check whether the ac-

tual outputs correspond to the expected ones. For in-

stance, test cases can be designed to validate whether

the observed behavior of the tested software conforms

to its speciﬁcation or not. This is mostly referred to in

the literature as Conformance testing. Dynamic Test-

ing can be used to verify different properties either

functional or non-functional.

• Functional Testing: can be classiﬁed into three

categories:

It is worthy noting that Ethereum supports several pro-

gramming languages (e.g., Vyper) and compilers.

https://solidity.readthedocs.io/

Static analysis can be used with formal veriﬁcation

methods with the aim to prove the correctness of a given

software.

ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering

356

– Black-box testing technique: it focuses on ex-

amining the software functionalities without

any knowledge of source code. It requires

the system speciﬁcation and it is usually per-

formed at the integration and system level. Typ-

ical black-box testing techniques include essen-

tially model-based testing (MBT) and fuzz test-

ing

(Klees et al., 2018).

– White-box testing technique: it is the detailed

investigation of internal logic and structure of

the code. This technique requires a full knowl-

edge of source code and it is usually applied at

unit testing level.

– Grey-box testing technique: it provides com-

bined beneﬁts of black-box and white-box test-

ing techniques. It requires a partial knowledge

of internal structure of the application.

• Non-functional Testing: aim to check non-

functional requirements, such as reliability, per-

formance and security.

In the case of testing blockchain oriented software, we

identify in the literature several kinds of testing tech-

niques that are performed with the aim of increasing

conﬁdence and trustworthiness of BoS. For instance,

we cite:

• Smart contract testing: It consists in performing

detailed functional testing of business logic and

process.

• Performance testing: It veriﬁes the performance

and the latency within the Blockchain network.

• Security testing: It performs security validation at

transaction, block, and network levels.

• Node testing: It aims at checking the integrity of

the network and the overall ledger by verifying

that all newly added transactions are saved ade-

quately.

3 RELATED SURVEYS

Veriﬁcation and validation of blockchain oriented

software have gained the attention of many re-

searchers. In this direction, we have identiﬁed re-

cent comprehensive surveys and systematic literature

reviews that focus on veriﬁcation of smart contracts.

For example, authors in (Liu and Liu, 2019) propose

an in-depth survey in which they study papers dealing

with the security assurance and papers related to the

correctness veriﬁcation (i.e., using formal techniques

Fuzzing is an automated testing technique that consists

in generating random data as inputs to a software.

such as model checking and theorem proving). More-

over, they summarize research done in this area and

point out new research directions in smart contract se-

curity and correctness.

Similarly, another recent survey, dated on 2020

(Tolmach et al., 2020), analyses and classiﬁes exist-

ing approaches to formal modeling, speciﬁcation, and

veriﬁcation of smart contracts. First, this work out-

lines various types of properties from different smart

contract domains (e.g., functional correctness and se-

curity). Second, it discusses the current approaches,

tools and frameworks used in verifying such prop-

erty speciﬁcations. It covers a wide range of pa-

pers dealing with model checking, theorem proving,

program veriﬁcation, symbolic execution and test-

ing. From testing perspective, it analyses only four

fuzzing-based approaches. Also, we notice that its

focus is on smart contract functional behavior and

does not discuss problems related to other blockchain-

related execution aspects, such as performance, scal-

ability, consensus, etc.

Authors in (S

anchez-G

omez et al., 2020) de-

velop a systematic literature review carried out in

the ﬁeld of smart contract development life cycle.

They review essentially the state of art of formal

smart contract modeling and code generation. A re-

duced number of studies dealing with testing BoS

were brieﬂy discussed. We have identiﬁed another

study (Koul, 2018) which underlines several chal-

lenges faced when testing BoS. It also acknowledges

the need to reuse specialized tools and techniques for

blockchain-oriented software testing with the aim of

ensuring high levels of quality. Nevertheless, the au-

thors do not address nor analyze the current research

in this topic. To the best of our knowledge, we have

not found recent surveys that provide comprehensive

studies related to software testing topic and identify

exhaustively the amount and quality of available re-

search results related to it. Therefore, our proposal

targets recent research work by analyzing and clas-

sifying approaches and tools in the area of software

testing of BoS, discussing their challenges as well as

the open research directions for future work.

4 TESTING BLOCKCHAIN

ORIENTED SOFTWARE

Although testing blockchain oriented software should

cover several layers : application layer (e.g., DApps),

smart contract layer, blockchain layer (e.g., blocks,

transactions), consensus layer and network layer, test-

ing efforts are concentrated essentially on testing

smart contracts and ensuring their correctness.

A Comprehensive Review of Testing Blockchain Oriented Software

357

A wide range of papers focus on security testing

and on detecting vulnerabilities with respect to the so-

lidity language. . Up to our knowledge, only (Yuan

and Zhu, 2020) proposes a novel approach that deals

with performance testing of blockchain system. The

proposed test tool measures several test metrics such

as delay, transaction execution success rate and re-

source consumption (i.e., CPU usage, memory usage,

etc.). Moreover, existing work on testing BoS can be

categorized into static testing approaches like and dy-

namic testing ones. Due to space limitation, we focus

in this review on studying only dynamic testing ap-

proaches.

4.1 White-box Testing Approaches

This research line is based on the investigation of in-

ternal logic and structure of the smart contract code.

Up to our knowledge, the majority of studied papers

deal with testing solidity source code and some of

them claim that their proposals can be extended to

other languages and platforms. For example, the au-

thors in (Driessen et al., 2021) present and partially

validate AGSOLT, a tool that creates test suites aim-

ing to achieve branch coverage for Solidity smart con-

tract unit testing. AGSOLT works with both a ran-

dom testing and a guided-search approaches. It was

tested on a set of real-world smart contracts from

GitHub. However, AGSOLT was not validated for

permissionned/commercial smart contracts.

A wide range of papers focus on mutation testing

(Li et al., 2019; Ivanova and Khritankov, 2020; Wang

et al., 2019b; Akca et al., 2019; Andesta et al., 2020;

Hartel and Schumi, 2020) and show that this testing

technique has a good effect on smart contract quality.

In fact, mutation testing is considered as a fault-based

software testing technique generally used to evaluate

the adequacy of test cases and their fault detection ca-

pabilities. In other words, its purpose is to help testers

in identifying limitations of the test suites.

In this direction, a well established approach is

proposed in (Li et al., 2019) providing a mutation test-

ing tool for Ethereum smart contracts called, MuSC.

This proposal takes as input a smart contract under

test and transforms its source ﬁles to Abstract Syn-

tax Tree (AST) version. Next, it generates various

mutants that implement traditional mutation opera-

tors and new ones according to the characteristics

of solidity language. The obtained mutants are then

transformed back to solidity source ﬁles with injected

faults for compilation, execution and testing purposes.

It also provides user-friendly interface to create test

nets and to display test reports. The latter include

execution results for each mutant (i.e., pass or fail)

and the total mutation score. Similarly, authors in

(Ivanova and Khritankov, 2020) developed a Regu-

larMutator tool for mutation analysis. Its major goal

is to improve the test suites in order to ﬁnd defects

as well as to increase the effectiveness and the fault

detection capabilities of test suites. Taken as input a

Trufﬂe project, RegularMutator generates mutants for

each source ﬁle in the project. Once mutant ﬁles are

generated, it substitutes the original ﬁles with the mu-

tant ones, executes project test suites, and then, the

test output is analysed. The main problem within this

approach is its high computational cost of executing a

set of tests when generating numerous mutants.

In (Wang et al., 2019b), the authors deal with the

problem of Ethereum smart contracts test generation,

targeting three main objectives, minimizing uncov-

ered branch coverage, time cost, and gas cost. They

also proposed a random based and an NSGA-II based

multi-objective approach to seek cost-effective test-

suites. The authors lead their empirical study on a set

of smart contracts in the most widely used Ethereum

decentralized applications and veriﬁed that their pro-

posed approaches could signiﬁcantly reduce both the

gas cost and the time cost while retaining the abil-

ity to cover branches. To ameliorate their work, the

authors need to conduct more experiments and to em-

ploy more optimization objectives as maximizing mu-

tation killed ability during the test-suite generation.

The authors present in (Akca et al., 2019) a fully

automated technique, SolAnalyser, for vulnerability

detection over Solidity smart contracts that uses both

static and dynamic analysis. They also implemented

a tool that injects different types of vulnerabilities in

smart contracts and used the mutated contracts for as-

sessing the effectiveness of different analysis tools.

The main contributions in this paper concern static

checks, developing of a test generator and a fault

seeding tool, run-time monitoring and empirical eval-

uation. The authors intend to generate inputs that pro-

vide control ﬂow coverage within functions and trans-

actions in Solidity contracts. This tool is considered

effective in terms of execution time due to the use

of AST code representation. In contrast to Regular-

Mutator (Ivanova and Khritankov, 2020), it did not

compare mutation operators for their effectiveness in

detection vulnerabilities and did not explore the pos-

sibility of using the results to improve tests.

In (Andesta et al., 2020), the authors propose a

testing mechanism for Solidity smart contracts based

on mutation testing. They analyzed a comprehen-

sive list of known bugs in such smart contracts and

designed classes of mutation operators inspired by

the real faults. In this work, the authors extended

the Universal Mutator tool with their proposed mu-

ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering

358

tation operators to automatically generate mutants

for smart contracts written in Solidity. In (Hartel

and Schumi, 2020), the authors evaluate the quality

of smart contract mutation testing at scale using re-

play tests that were automatically generated from the

Ethereum blockchain. They propose a set of muta-

tion operators on the basis of existing works and eval-

uate these operators at scale. The authors introduce

a novel killing condition based gas limits for smart

contract transactions aiming to improve the mutation

score. A ﬁrst limitation of this work concerns a threat

to the validity of the proposed evaluation in the way

that the authors only consider a replay test suite that

is less powerful than other testing techniques, which

might get a higher mutation score. Another limita-

tion regarding the validity of the proposed approach

might be that it is not wise to kill a mutant only based

on a different gas usage since it could still be seman-

tically equal. Even it is widely considered as an ef-

fective method of enhancing the adequacy of testing,

mutation testing is challenging and hard to be used in

industry due to its high cost. It requires a high under-

standing of the semantics of the smart contract lan-

guage to generate only useful mutants.

Another promising research line is identiﬁed

that consists in testing Decentralized Applications

(DApp). A DApp is a Web application composed of

two parts: the front-end and the back-end. The iden-

tiﬁed approaches (Gao et al., 2019; Wu et al., 2020)

touch several research areas including smart contract

analysis and automated Web application testing. They

overcome the lack of effective methods and tools for

testing DApps since the existing ones either focus on

testing front-end code or back-end programs but they

ignore the interaction between them. For instance, au-

thors in (Gao et al., 2019) propose a tool called Sun-

gari that employs ﬁrst random events to infer the in-

teraction between browser-side code and smart con-

tracts. Second, the proposed technique generates test

cases and orders them based on a data ﬂow graph

of smart contracts. In the same direction, the Kaya

framework cited in (Wu et al., 2020) reduces the com-

plexity of testing DApps by providing GUI and CLI

tool. It consists of three modules: a Kaya Graph-

ical User Interface (KGUI), Core Function Module

(CFM) and Log Analyzer (LA). Within Kaya, test en-

gineers may produce easily test cases with a simple

setting and then Kaya will execute them in an auto-

matic manner. This is done by simulating front-end

events, setting Blockchain pre-states (i.e., runtime en-

vironment of test cases) and ﬁnally running smart

contracts. The Log Analyzer compares the initial val-

ues of parameters in SCs and the generated outputs,

then, it produces a report illustrating such analysis.

4.2 Black-box Testing Approaches

As already mentioned, Black-box testing includes

several kinds of testing techniques that apply testing

activities without having any knowledge of the inter-

nal structure of BoS. The most used ones in the stud-

ied context are fuzz testing and model-based testing

(Krichen, 2007; Krichen, 2018).

From fuzz testing perspective, we identify the

Fuse project (Chan and Jiang, 2018) which is a fuzz

testing service for smart contracts and Dapp testing.

It assists developers for test diagnosis via test sce-

nario visualization. The ﬁrst prototype developed in

the context of Fuse project is ContractFuzzer (Jiang

et al., 2018) that detects seven security vulnerabili-

ties of Ethereum smart contracts. The proposed ap-

proach generates fuzzing inputs from the ABI

spec-

iﬁcation of the smart contract. It also deﬁnes test or-

acles for detecting the supported real world vulner-

abilities within smart contracts. ContractFuzzer was

performed on 6991 real-world Ethereum SCs showed

that it has identiﬁed 459 SCs vulnerabilities, includ-

ing the DAO and Parity Wallet attacks.

A closely approach to ContractFuzzer is sFuzz,

an adaptive fuzzing engine for EVM smart contracts

(Nguyen et al., 2020). sFuzz is composed of three

components: runner that manages test case execu-

tion, liboracles that supports eight oracles inspired

by the previous researches (Jiang et al., 2018; Luu

et al., 2016) and libfuzzer which implements the test

suite generation algorithm. The latter is based on

a feedback-guided fuzzing technique which trans-

forms the test generation problem into an optimiza-

tion problem and uses feedbacks as an objective func-

tion in solving the optimization problem. This pro-

posal is based on adaptive strategy since it is possible

to change the objective function adaptively based on

the feedback to evolve the test suite with the aim of

improving its branch coverage. Due to its effective-

ness and its reliability, sFuzz has already gained in-

terest from multiple companies and research organi-

zations. However, fuzz-based approaches may suffer

from false positive detection as a reported vulnerabil-

ity may be a false positive

From the model-based testing perspective, authors

in (Liu et al., 2020) introduce an MBT tool, namely

ModCon, that uses an explicit abstract model of the

target SC in order to derive tests automatically. Mod-

Con shows its effectiveness speciﬁcally for enter-

prise SC applications written in Solidity from permis-

sioned/consortium blockchains. It allows SC devel-

Application Binary Interface

Some test cases fail but there is no bug and the program

is working correctly.

A Comprehensive Review of Testing Blockchain Oriented Software

359

opers to input their test model for the SC under test.

Certainly, the efﬁciency of MBT depends both on the

input test model and fault model (or test hypotheses).

A promising research challenge could be to investi-

gate SCs and their faults to determine suitable fault

models or test assumptions. In the same direction,

authors in (S

anchez-G

omez. et al., 2019) propose a

model driven approach that generates smart contract

code from UML diagrams (i.e., Use Cases and Ac-

tivity diagrams). They also point out the necessity of

applying testing technique in the early stage of Soft-

ware Development Life Cycle (SDLC), especially in

the context of blockchain oriented software. It out-

lines the necessity of applying model-based testing

process to generate test scenarios from requirements

(UML use cases) and test cases from UML activity

diagrams. However, this approach is still immature

since no transformation rules for the smart contract

generation code are given and no real tool implemen-

tation for the discussed ideas were presented. Simi-

larly, the work in (Kakadiya, 2017) proposes a com-

plete software testing life cycle to test BoS projects.

The proposal is composed of four phases including

system overview, test design, test planning and test

execution.

4.3 Grey-box Testing Approaches

Another research line is identiﬁed. It consists in ap-

plying testing techniques with a partial knowledge of

internal BoS structure. In this direction, the work

of Wang et al. (Wang et al., 2019a) combines fuzz

and mutation testing in order to detect several smart

contract vulnerabilities triggered during transactions.

The proposed tool called ContraMaster is a grey-box

fuzzing approach that dynamically executes transac-

tions and observes their actual effects on the contract

states in order to detect vulnerabilities.

ContraMaster includes a novel feedback mech-

anism to guide efﬁciently the fuzzing process, by

taking into consideration the data-ﬂow information,

control-ﬂow information and states of smart con-

tracts. It also monitors the executions of target smart

contracts, and validates the obtained results against a

general purpose semantic test oracle to detect vulner-

abilities. Compared to traditional fuzzing approaches

like ContractFuzz (Jiang et al., 2018), this proposal

has not shown any false positives, and it easily gener-

alizes to unknown types of vulnerabilities.

HARVEY (W

ustholz and Christakis, 2019) is an-

other grey-box Fuzzer for smart contracts, which is

the ﬁrst one used in industry by one of the largest

blockchain-security consulting companies. It is based

on a novel technique that systematically forecasts new

inputs for the smart contract under test in order to in-

crease the performance and effectiveness of the grey-

box fuzzing approach. In contrast to existing works

based on grey-box fuzzing, this proposal suggests

concrete input values based on data from past execu-

tions, instead of performing arbitrary mutations.

Similar to ContraMaster tool, HARVEY gener-

ates, executes, and fuzzes sequences of transactions,

which invokes the contract’s functions. To allevi-

ate the exploration of the search space of all possi-

ble sequences, authors use a demand-driven sequence

fuzzing technique, avoiding the generation of transac-

tion sequences that cannot further enhance the cover-

age.

Another promising study was proposed in (Liao

et al., 2019). This work introduces SoliAudit tool that

combines machine learning and Fuzz testing for smart

contract vulnerability assessment. Indeed, it em-

ploys machine learning using Solidity machine code

as learning features to verify 13 kinds of vulnerabili-

ties. The latter are listed as Top 10 threats by an open

security organization

Moreover, the tool includes a grey-box fuzz test-

ing mechanism, composed of a fuzzer contract and a

simulated blockchain environment for on-line trans-

action veriﬁcation. Similar to (Wang et al., 2019a),

SoliAudit can detect vulnerabilities without prede-

ﬁned patterns. The source code is only required if the

user wants to identify the vulnerability location in the

source code. It only needs the ABI and the bytecode

to deploy on the blockchain.

Yet another potential work to explore is the HF-

ContractFuzzer tool (Ding et al., 2021). Compared to

the majority of cited studies that focus on Ethereum

smart contracts written in solidity language, this tool

is speciﬁc to a Hyperledger Fabric smart contract

written in Go language. Indeed, authors deal with

the security vulnerability detection while combining

Fuzzing technology and Go language testing tool,

called go-fuzz.

To do so, HFContractFuzzer takes as input smart

contracts and test cases. Then, it automatically gen-

erates test results. In order to improve the efﬁciency

of smart contract fuzzing, this proposal gives two op-

timization methods of the go-fuzz tool : optimizing

the generation of the initial Fuzzing corpus and op-

timizing the mutation process. However, it requires

more effort to evaluate its vulnerability detection and

to measure its performance.

DASP Top 10: https://dasp.co/

ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering

360

5 CHALLENGES AND FUTURE

DIRECTIONS

5.1 Challenges

1. Smart Contract Testing still a hard task for devel-

opers: Despite the growing interest generated by

its use in a wide range of applications, the use and

the validation of smart contract technology is still

considered as tough mission for many developers,

mainly because of its innovative design.

2. Formal methods for smart contract testing still

overlooked: Formal testing is regarded as a crucial

part of the software development process. Formal

testing, on the other hand, has been largely dis-

regarded in the smart contract development cycle.

Although a shift of formal testing from standard-

ised software to smart contract-based software is

quite conceivable.

3. Performance tools still not enough evaluated: De-

spite the fact that several techniques for evalu-

ating Blockchain performance have been devel-

oped, only a few of them have been thoroughly ex-

amined due to the lack of standardised interfaces

while running workloads.

4. Choice of performance metrics: Many developers

utilize various types of coverage to demonstrate

the effectiveness of testing. Although this is an

intuitive indicator, there is no inescapable link be-

tween increased coverage and improved vulnera-

bility detection efﬁciency.

5.2 Future Directions

1. Making Smart Contract Testing easier: The most

important thing regarding this point is to deﬁne

some speciﬁc techniques which may enable de-

velopers to establish a testing procedure dedicated

for smart contracts.

2. Formal methods for smart contract testing: At this

level too, there is a real need to make formal test-

ing techniques more accessible for smart contracts

developers by providing more sophisticated tools

which are both efﬁcient and easy to use at the

same time.

3. Performance tools: these tools must evolve in or-

der to meet the need of developers and to meet

high-quality levels. In particular, they need to be

equipped with standardised interfaces which may

be used while running workloads.

4. Choice of performance metrics: At this level, new

criteria and performance metrics have to be de-

ﬁned in order to improve the efﬁciency of vulner-

ability detection.

6 CONCLUSION

The main motivation for this study was to investigate

the state of art in testing Blockchain oriented Soft-

ware. This review was conducted to determine what

issues have been studied and by what means, also to

provide a guide for researchers in this emergent re-

search area. Up to our best knowledge, the existing

surveys covered essentially static testing and analysis.

Dynamic testing approaches were rarely addressed.

Therefore, this paper is a signiﬁcant contribution in

the literature which includes over 20 research stud-

ies published from 2017 to 2021. A classiﬁcation of

these approaches was reported while providing their

strengths and weaknesses. In summary, the obtained

results of this study indicated that most research pa-

pers are from conference proceedings, suggesting that

the area of research is still young. The proposed re-

search direction may lead to advances in three rela-

tively open ﬁelds of software engineering research:

(1) Model based testing of BoS, (2) Performance test-

ing of blockchain networks and (3) Testing consensus

algorithms. We believe that to improve the current

state of art in these three lines, further work needs to

be done and more exploration is required.

REFERENCES

Akca, S., Rajan, A., and Peng, C. (2019). Solanalyser: A

framework for analysing and testing smart contracts.

In 2019 26th Asia-Paciﬁc Software Engineering Con-

ference (APSEC), pages 482–489.

Ali, M. S., Vecchio, M., Pincheira, M., Dolui, K., An-

tonelli, F., and Rehmani, M. H. (2018). Applications

of blockchains in the internet of things: A comprehen-

sive survey. IEEE Communications Surveys & Tutori-

als, 21(2):1676–1717.

Andesta, E., Faghih, F., and Fooladgar, M. (2020). Testing

smart contracts gets smarter. In 2020 10th Interna-

tional Conference on Computer and Knowledge Engi-

neering (ICCKE), pages 405–412.

Antonino, P. and Roscoe, A. (2020). Formalising and veri-

fying smart contracts with solidiﬁer: a bounded model

checker for solidity. arXiv preprint arXiv:2002.02710.

Chan, W. and Jiang, B. (2018). Fuse: An architecture for

smart contract fuzz testing service. In The 25th Asia-

Paciﬁc Software Engineering Conference (APSEC),

pages 707–708.

Ding, M., Li, P., Li, S., and Zhang, H. (2021). Hfcontract-

fuzzer: Fuzzing hyperledger fabric smart contracts for

vulnerability detection. CoRR, abs/2106.11210.

A Comprehensive Review of Testing Blockchain Oriented Software

361

Driessen, S., Nucci, D. D., Monsieur, G., and van den

Heuvel, W.-J. (2021). Agsolt: a tool for automated

test-case generation for solidity smart contracts.

Fekih, R. B. and Lahami, M. (2020). Application of

blockchain technology in healthcare: A comprehen-

sive study. In The Impact of Digital Technologies on

Public Health in Developed and Developing Coun-

tries - 18th International Conference, ICOST 2020,

Hammamet, Tunisia, June 24-26, 2020, Proceedings,

pages 268–276.

Finley, K. (2016). A $50 million hack just showed that the

dao was all too human.

Freedman, R. (1991). Testability of Software Compo-

nents. IEEE Transactions on Software Engineering,

17(6):553 –564.

Gao, J., Liu, H., Li, Y., Liu, C., Yang, Z., Li, Q., Guan,

Z., and Chen, Z. (2019). Towards automated test-

ing of blockchain-based decentralized applications. In

IEEE/ACM 27th Int. Conf. on Program Comprehen-

sion (ICPC), pages 294–299.

Hartel, P. and Schumi, R. (2020). Mutation testing of smart

contracts at scale. In Ahrendt, W. and Wehrheim, H.,

editors, Tests and Proofs, pages 23–42, Cham.

Ivanova, Y. and Khritankov, A. (2020). Regularmutator:

A mutation testing tool for solidity smart contracts.

Procedia Computer Science, 178:75–83.

Jabbar, R., Dhib, E., ben Said, A., Krichen, M., Fetais, N.,

Zaidan, E., and Barkaoui, K. (2022). Blockchain tech-

nology for intelligent transportation systems: A sys-

tematic literature review. IEEE Access.

Jiang, B., Liu, Y., and Chan, W. K. (2018). Contractfuzzer:

Fuzzing smart contracts for vulnerability detection.

ASE 2018, page 259–269.

Kakadiya, A. (2017). Block-chain oriented software test-

ing approach. International Research Journal of En-

gineering and Technology (IRJET).

Klees, G., Ruef, A., Cooper, B., Wei, S., and Hicks, M.

(2018). Evaluating fuzz testing. In Proceedings of

the 2018 ACM SIGSAC Conference on Computer and

Communications Security, page 2123–2138.

Koul, R. (2018). Blockchain oriented software testing -

challenges and approaches. In 3rd International Con-

ference for Convergence in Technology (I2CT), pages

1–6.

Krichen, M. (2007). Model-based testing for real-time sys-

tems. PhD thesis, PhD thesis, PhD thesis, Universit

Joseph Fourier (December 2007).

Krichen, M. (2018). Contributions to model-based test-

ing of dynamic and distributed real-time systems.

PhD thesis,

Ecole Nationale d’Ing

enieurs de Sfax

(Tunisie).

Li, Z., Wu, H., Xu, J., Wang, X., Zhang, L., and Chen, Z.

(2019). Musc: A tool for mutation testing of ethereum

smart contract. In 2019 34th IEEE/ACM Interna-

tional Conference on Automated Software Engineer-

ing (ASE), pages 1198–1201.

Liao, J.-W., Tsai, T.-T., He, C.-K., and Tien, C.-W. (2019).

Soliaudit: Smart contract vulnerability assessment

based on machine learning and fuzz testing. In

Sixth International Conference on Internet of Things:

Systems, Management and Security (IOTSMS), pages

458–465.

Liu, J. and Liu, Z. (2019). A survey on security veriﬁcation

of blockchain smart contracts. IEEE Access, 7:77894–

77904.

Liu, Y., Li, Y., Lin, S.-W., and Yan, Q. (2020). Modcon: A

model-based testing platform for smart contracts. In

Proceedings of the 28th ACM Joint Meeting on Eu-

ropean Software Engineering Conference and Sym-

posium on the Foundations of Software Engineering,

page 1601–1605.

Luu, L., Chu, D.-H., Olickel, H., Saxena, P., and Hobor, A.

(2016). Making smart contracts smarter. In Proceed-

ings of the 2016 ACM SIGSAC Conference on Com-

puter and Communications Security, page 254–269.

Nakamoto, S. et al. (2008). Bitcoin: A peer-to-peer elec-

tronic cash system.

Nelaturu, K., Mavridou, A., Veneris, A., and Laszka, A.

(2020). Veriﬁed development and deployment of

multiple interacting smart contracts with verisolid.

In Proc. of the 2nd IEEE International Conf. on

Blockchain and Cryptocurrency (ICBC).

Nguyen, T. D., Pham, L. H., Sun, J., Lin, Y., and Minh, Q. T.

(2020). Sfuzz: An efﬁcient adaptive fuzzer for solid-

ity smart contracts. In Proceedings of the ACM/IEEE

42nd International Conference on Software Engineer-

ing, page 778–788.

Praitheeshan, P., Pan, L., Yu, J., Liu, J. K., and Doss,

R. (2019). Security analysis methods on ethereum

smart contract vulnerabilities: A survey. CoRR,

abs/1908.08605.

anchez-G

omez., N., Morales-Trujillo., L., and Torres-

Valderrama., J. (2019). Towards an approach for ap-

plying early testing to smart contracts. In Proceedings

of the 15th International Conference on Web Infor-

mation Systems and Technologies - APMDWE,, pages

445–453.

anchez-G

omez, N., Torres-Valderrama, J., Garc

ıa-Garc

ıa,

J. A., Guti

errez, J. J., and Escalona, M. J.

(2020). Model-based software design and testing in

blockchain smart contracts: A systematic literature re-

view. IEEE Access, 8:164556–164569.

Tolmach, P., Li, Y., Lin, S., Liu, Y., and Li, Z. (2020). A

survey of smart contract formal speciﬁcation and ver-

iﬁcation. CoRR, abs/2008.02712.

Wang, H., Li, Y., Lin, S.-W., Artho, C., Ma, L., and Liu,

Y. (2019a). Oracle-supported dynamic exploit gener-

ation for smart contracts.

Wang, X., Wu, H., Sun, W., and Zhao, Y. (2019b). To-

wards generating cost-effective test-suite for ethereum

smart contract. In IEEE 26th International Confer-

ence on Software Analysis, Evolution and Reengineer-

ing (SANER), pages 549–553.

Wu, Z., Zhang, J., Gao, J., Li, Y., Li, Q., Guan, Z., and

Chen, Z. (2020). Kaya: A testing framework for

blockchain-based decentralized applications. In IEEE

International Conference on Software Maintenance

and Evolution (ICSME), pages 826–829.

ustholz, V. and Christakis, M. (2019). Harvey: A greybox

fuzzer for smart contracts. CoRR, abs/1905.06944.

Yuan, C. and Zhu, J. (2020). A new performance testing

scheme for blockchain system. In Zhang, J., Dres-

ner, M., Zhang, R., Hua, G., and Shang, X., editors,

LISS2019, pages 757–773.

ENASE 2022 - 17th International Conference on Evaluation of Novel Approaches to Software Engineering

362