AN APPROACH TO METAMORPHIC TESTING FOR WS-BPEL

COMPOSITIONS

Carmen Castro-Cabrera and Inmaculada Medina-Bulo

Department of Computer Languages and Systems, University of C´adiz, C´adiz, Spain

Keywords:

Testing, Metamorphic testing, Mutation analysis, Oracle, Follow-up test cases, Web services, WS-BPEL.

Abstract:

Nowadays, Web Service (WS) compositions play an important role in business processes. Languages for com-

posing web services, such as the OASIS WS-BPEL 2.0 standard, open a new ﬁeld for large-scaleprogramming.

However, they also present a challenge for traditional quality assurance because of the inclusion of speciﬁc

instructions for concurrency, fault compensation, and dynamic service discovery and invocation. Metamor-

phic Testing (MT) has proved useful to test and improve the quality of traditional imperative programs. This

paper discusses how to use MT to test WS compositions in WS-BPEL. Although MT has not been previously

applied to this area, promising results have been obtained in a number of different applications. A component

diagram for a testing framework implementing this approach is included.

1 INTRODUCTION

Web services are having a strong impact in our so-

ciety because of the increasing number of Internet

transactions and the new ways of conceiving and us-

ing web applications dealing with them. The Web

Services Business Process Execution Language (WS-

BPEL), (OASIS, 2007) allows us to develop new Web

Services (WS) by modeling more complex business

processes on top of preexisting WS. This evolution

of business processes has produced the development

of speciﬁc software to satisfy this demand. However,

this development has not come with an advance in

testing for this kind of software. In addition, the eco-

nomic impact of WS-BPEL service compositions has

increased (IDC, 2008), and deeper insight on how to

test them effectively is therefore required. We present

a proposal to apply MT to WS-BPEL compositions.

MT (Chen, 1998) is a software testing technique

using metamorphic relations (MR). MR are existing

or expected relations deﬁned on a set of inputs and

their corresponding outputs for multiple executions of

a function under test. The underlying concept is sim-

ple and its automation is not difﬁcult. In fact, it has

proved successful in testing and improving the qual-

ity of traditional imperative programs (Zhou et al.,

2004). Regarding the cost effectiveness of MT, Zhang

(Zhang et al., 2009) conducted an experiment where

the fault detection capabilities and time cost of MT

were compared to the standard assertion checking me-

thod. Results show that MT has the potential to detect

more faults than the assertion checking method. In

addition, Chan and his collaborators are investigating

their applications to different areas. The most related

to this paper are the application of MT to service-

oriented software (Chan et al., 2006; Chan et al.,

2007).

The structure of the rest of the paper is as follows:

Section 2 outlines the main concepts about service

compositions and MT. Section 3 presents some tech-

niques that are being applied with success to testing

WS compositions. In the following Section 4, some

applications of MT to different areas are reviewed.

Section 5 presents our proposal, its main steps and

a component diagram for applying MT to WS-BPEL

compositions. Finally, Section 6 presents some con-

clusions and future research lines.

2 PRELIMINARIES

Next, we introduce the main concepts about service

compositions in WS-BPEL and MT.

2.1 Web Services Compositions

This paper focuses in WS compositions, specially

when they are implemented using WS-BPEL. WS-

BPEL is a programming language based in XML that

137

Castro-Cabrera C. and Medina-Bulo I..

AN APPROACH TO METAMORPHIC TESTING FOR WS-BPEL COMPOSITIONS.

DOI: 10.5220/0003611401370142

In Proceedings of the International Conference on e-Business (ICE-B-2011), pages 137-142

ISBN: 978-989-8425-70-6

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

is used to generate business processes from preexist-

ing services. The resulting business process can be

then reused as a WS in higher level compositions. A

WS-BPEL composition contains four sections:

1. Declarations of the relationships to the external

partners. These include both the client that has in-

voked the business process and the external part-

ners whose services are required to complete the

request of the client.

2. Declarations of the variables used by the process

and their types. Variables are used for storing both

the messages received and sent from the business

process and the intermediate results required by

the internal logic of the composition.

3. Declarations of handlers for various situations,

such as fault, compensation or event handlers.

4. Description of the business process behavior.

The major building blocks in WS-BPEL are the

activities. Activities may have both attributes and a

set of containers. These containers can also include

elements with their own attributes. Here is an exam-

ple:

<flow>

← Structured activity

<links>

← Container

← Element

</links>

<invoke name="checkFlight"

...

← Basic activity

← Container

← Element

</sources>

</invoke>

<invoke name="checkHotel"

...

<invoke name="checkRentCar"

...

<invoke name="bookFlight"

← Attribute ...

← Container

</targets>

</invoke>

</flow>

WS-BPEL provides concurrency and synchro-

nization primitives. For instance, the

flow

activ-

ity runs a set of activities in parallel. Synchroniza-

tion constraints between activities can be deﬁned. In

the above example, the

flow

activity invokes three

WS in parallel:

checkFlight

checkHotel

, and

checkRentCar

. There is another WS,

bookFlight

that will only be invoked if

checkFlight

is com-

pleted. Activities are synchronized by linking them:

the target activity of every link will only be executed

if the source activity of the link has been completed

successfully.

2.2 Metamorphic Testing

Software testing is a key activity in any software

development project. Fault detection and correc-

tion are key activities to ensure program reliability.

For this reason, a number of software testing tech-

niques have been developed following different ap-

proaches (Beizer, 1990; Myers, G.J. et al., 2004).

One of the main challenges for most testing tech-

niques is the oracle problem. Because of its nature,

some programs are inherently difﬁcult to test. For ex-

ample, the output can be difﬁcult to verify because the

precise result is not known a priori,

or because the

size of the output makes it unfeasible to apply stan-

dard techniques. In these cases, we should use a dif-

ferent approach to help with the veriﬁcation and vali-

dation process.

Traditionally, human testers have been used to

check the results manually, though it is clear that

this is both expensive and error-prone. An alterna-

tive is to develop a simpler program implementing the

same functional requirements that the original pro-

gram, sacriﬁcing efﬁciency and other non-functional

requirements. However, this is out of the question for

complex systems and reduces the problem to the cor-

rectness of the simpler program. Any mechanism for

checking whether a program under test behaves cor-

rectly for any given input is called an oracle. As we

have argued, there are practical situations where or-

acles can be unavailable or too expensive to apply.

MT has been proposed to alleviate the oracle prob-

lem (Chen, 1998; Chen, 2010).

MT relies on the notion of metamorphic relation

(MR). In (Andrews et al., 2005), they are deﬁned as

“existing or expected relations over a series of dis-

tinct inputs and their corresponding results for mul-

tiple evaluations of a target function”. When the

implementation is correct, program inputs and out-

puts are expected to satisfy some necessary proper-

ties that are relevant for the underlying algorithms.

These properties are traditionally known as asser-

tions. In these sense, a MR is a kind of assertion.

However, a MR should provide a way of generating

new test cases from given ones. In order to illus-

trate this, let us consider a sorting program sort that

sorts an input list of integers producing a list with the

same integers in ascending order. For example, when

the list l

= h4, 7, 2i is used, the expected result of

sort(l

) is h2, 4, 7i. Then, if we use the permutation

= perm(l

, h(1, 2)i) = h7, 4, 2i as the follow-up test

case, the expected result is the same. In other words,

sort(l

) = sort(l

), obviously because any permuta-

tion of a list has the same ordered list. Therefore, we

could formalize this property, MR

, as follows:

≡ (∃x l

= perm(l

, x)) → sort(l

) = sort(l

)

Partial knowledge or properties of the result can, in-

deed, be available.

ICE-B 2011 - International Conference on e-Business

138

where l

is the original input, l

is a follow-up test

case (a permutation of l

in this particular case) and

perm is the function applying a permutation to a list

(just one swap in the example; the ﬁrst and the second

elements). If the metamorphic property is not satis-

ﬁed then the program is faulty.

Please, notice that perm is the test generation

function associated to MR

. It receives an input list

(the given test case) and a permutation (a list of in-

dex pairs), and produces a new list (the follow-up test

case). Test case generation can be automated as in tra-

ditional techniques. For example, replacing the sec-

ond parameter of perm by a random permutation we

would obtain the equivalentto traditional random test-

ing for this example. Of course, a single MR is gen-

erally insufﬁcient. In the above example, we could

not detect certain faults just with MR

, as correctness

for sorting implies permutation preservation (the re-

sulting list must be a permutation of the original) and

an order constraint (it must be sorted). On the other

hand, a MR is usually devised by an expert in the ap-

plication domain.

Summing up, MT is a testing technique using

MR (Chan et al., 2007). It begins with an initial test

suite, which is produced with any test case selection

strategy, and a set of MR. Once the program is exe-

cuted on the test suite, errors are ﬁxed until a success-

ful test suite (i.e., consisting of non-failing test cases)

is obtained. Then, MR are used to generate follow-

up test cases. These test cases form the follow-up test

suite and the process is iterated until the speciﬁed test

criteria are met.

3 TESTING WEB SERVICES

COMPOSITIONS

In this section, we describe some tools and techniques

applied to test this kind of software. We focus in those

which are implemented in WS-BPEL, specially the

automated ones. Most of them are referred in bib-

liography to test case generation of those composi-

tions (Garc´ıa-Fanjul et al., 2007; Yan et al., 2006;

Zheng et al., 2007). There are more complete works

including the measure of test case quality and opti-

mizing test case generation through mutation testing.

GAmera (UCASE Research Group, 2010a) is a mu-

tation testing framework to WS-BPEL that uses ge-

netic algorithms to reduce the number of mutants re-

quired. It classiﬁes mutants as killed (output is dif-

ferent than original composition), alive (output is the

same as the original composition against the entire

test case set performed), equivalents (output is al-

ways the same as the original composition), and er-

roneous (result in failure of the unfold) (Dom´ınguez-

Jim´enez et al., 2009). Other testing framework called

Takuan (UCASE Research Group, 2010b) is based on

dynamic generation of invariants for WS-BPEL com-

positions. Lastly, a recent work describes different

technologies developed in web services and web ser-

vices testing (Bozkurt et al., 2010).

4 METAMORPHIC TESTING IN

SOME ENVIRONMENTS AND

PROGRAMS

The ﬁrst documented work about MT belongs to

Weyuker (Weyuker, 1982). In this paper she pro-

posed a new perspective for software testing based on

using alternative programs to the original code shar-

ing the same objective to prove the original program.

However, she only focused on numerical functions

and the relations were of equality, such as (a+ b)

+ 2a∗ b+ b

That original notion was later adopted by

Chen (Chen, 1998), who deﬁned the term as Meta-

morphic Testing and extended it, including non-

numerical functions to be tested and MR but not nec-

essarily equality as previously.

Since then, other works have been written, each

one focusing on one or more different aspects of the

theory. For example, (Zhou et al., 2004) describes di-

verse programs to apply MT. (Gotlieb and Botella,

2003) relates to automatization of this technique and

where it is applicable. Moreover, a research group

from Columbia University has implemented part of

the process to apply MT using JML execution envi-

ronmental (Murphy et al., 2009). Furthermore, Chen

and his colleagues in (Chen et al., 2004) describe

the importance of choosing MR suitable with prob-

lem domain and algorithm structure of the program to

prove. In addition, Chan and his collaborators have

different approaches about this subject. Although, the

most interesting for us is how to apply MT to SOA

software (Chan et al., 2006; Chan et al., 2007). Fur-

thermore, there is other interesting issue based on the

analysis of feature model to obtain MR and automate

the test data generation (Segura et al., 2010).

4.1 MT with Numerical Problems

MT may be applied to resolve issues related to ora-

cle problems with partial differential equations where

the solution is unknown. A related study case (Chen

et al., 2002) addresses how to successfully apply MT

techniques to a thermodynamic problem to obtain ce-

AN APPROACH TO METAMORPHIC TESTING FOR WS-BPEL COMPOSITIONS

139

rtain properties representing MR.

4.2 MT with Non-numerical Problems

This technique is not limited to numerical problems

and may be applied to programs designed to solve

non-numerical problems (Zhou et al., 2004). For

example, it has been proven in programs implement-

ing algorithms that solve graph problems, with known

properties that may be proven and must be met. MT

has also been applied in computer graphics software.

Additionally it may be used in testing compilers when

it becomes too difﬁcult to prove the equivalence be-

tween source and object code.

Another approach is to use interactive software

where inputs to program are user inputs, instead of

data. As example, test cases to testing a browser are

HTML ﬁles and action sequences like this:

EnterURL → ClickItem1 →

SelectMenufile → SelectOpen...

There may be several action sequences for a spe-

ciﬁc user deﬁned requirement. By relating all inputs

to their suitable outputs for every matching action we

obtain MR. If we select the appropriate MR set we

may apply MT to this kind of software.

4.3 MT with Learning Machine

Programs

A group of Columbia University have implemented

a framework (Murphy et al., 2008) automating part

of the testing process. They have analyzed programs

that implement learning machines applications to au-

tomize how to obtain properties for this type of soft-

ware. In (Murphy et al., 2008) categorized six types

of metamorphic properties that such applications may

have. Then implemented a tool called Corduroy to

automate the process by allowing developers to spec-

ify individual functions of metamorphic properties us-

ing the speciﬁcation language JML; these properties

could then be checked using JML Runtime Assertion

Checking (Murphy et al., 2009). This way they ob-

tain and check Metamorphic Relations in order to ap-

ply MT.

5 AN APPROACH OF MT IN WEB

SERVICES COMPOSITIONS

Once the different testing aspects to web services and

MT have been analized, we propose to apply MT

to web services compositions in WS-BPEL and im-

plement MT by integrating well-tested open-source

systems: ActiveBPEL as the WS-BPEL engine and

BPELUnit as the unit test library. ActiveBPEL is a

WS-BPEL 2.0 compliant open-source engine. Com-

pared to other engines, it is quite lightweight, reduc-

ing the time needed to run a test suite. It is main-

tained by ActiveVOS (ActiveVOS, 2009), which of-

fers commercial products based on it. BPELUnit is a

WS-BPEL unit test library (Mayer and L¨ubke, 2006)

which can use any WS-BPEL 2.0 compliant engine. It

uses XML ﬁles to describe test suites. It can replace

external services with mockups providing predeﬁned

responses.

Our approach is based in the two following ideas:

1. Based in Murphy and his collaborators (Murphy

et al., 2008; Murphy et al., 2009), the ﬁrst idea

consists of achieving program properties from

learning machines software, automating MR se-

lection.

2. The second idea is to specify, design and imple-

ment a new web service called metamorphic ser-

vice that wraps the service to prove with MR. This

approach is proposed by Chan and his colleagues

in (Chan et al., 2006; Chan et al., 2007)

A generic diagram of our approach is as follows:

Firstly, a metamorphic composition will be speciﬁed

to encapsulate each composition to be proven, to in-

clude MR previously selected by analyzing the com-

position to obtain properties. This way it will allow

us to automate part of the process. Furthermore, MR

will allow to generate follow-up test cases and prove

results sound respect properties and, therefore, prove

the original composition. Finally, metamorphic com-

position will issue a report with results to be analyzed.

In our proposal (Fig. 1) is necessary for each compo-

sition:

• Select adequate MR.

• Generate initial test cases.

• Get follow-up test cases.

• Multiple executions of composition.

• Analyze results.

To begin, we take a composition example. We

comment brieﬂy this approach with the classical WS-

BPEL example of the Loan ApprovalService included

in the WS-BPEL 2.0 speciﬁcation (OASIS, 2007). To

begin, we have to develop a program to ingest a ﬁle

containing a BPELUnit Test Suite (BPTS) using tem-

plates to generate initial test cases and return other

BPTS with follow-up test cases (from initial ones ap-

plying MR previously selected). To achieve this, we

will do the following:

ICE-B 2011 - International Conference on e-Business

140

Analize composition to obtain properties

tecnique

Obtain the initial (succesful) test cases with other

Design and implement suitable MR

Specify, design and implement a metamophic

adding MR

Compare results and metamorphic composition

Apply MR to obtain follow−up test cases

composition based on initial composition,

Execute initial test cases and follow−up test cases

issues to obtain a report

Figure 1: A component diagram.

• Hand write a BPTS for the example composition

using templates. This is already completed.

• Write the data ﬁle to generate the initial test cases

by the BPTS. This is completed as well. It is a

CSV ﬁle.

• Design the MR.

• Implement a program that receives the data

ﬁle(CSV) and generates a new ﬁle (CSV) using

BPTS to generate one or more test cases using the

MR designed. We have to associate the initial test

cases with the new ones.

• Execute the sample composition with the initial

and new test cases.

• Analize and compare the results in both execu-

tions.

Another option would be to develop a program to

ingest a BPTS (initial test cases written by hand) and

return other BPTS once MR are applied. by complet-

ing the following:

• Load BPTS in memory (this would be done by

BPELUnit).

• Conver the BPTS in memory using MR.

• Use XMLBeans to save the new BPTS from mem-

ory to a ﬁle in disk.

We are currently working on the ﬁrst option of the

proposed approach. The proposed Metamorphic com-

position includes all the tools and ﬁles used to gener-

ate initial and follow-up test cases, as well as, MR

code implemented besides original composition.

6 CONCLUSIONS AND FUTURE

WORK

Business processes based in WS-BPEL compositions

are rapidly becoming commonplace in recent years,

so it is important to devote more attention to testing

in this context.

Therefore, it is required more innovative and efﬁ-

cient techniques in software testing to WS composi-

tions.

The WS-BPEL programming language is oriented

to business processes, and therefore its peculiarities

must be considered. MT is proven efﬁcient on dif-

ferent applications, and research by several groups

continues. Moreover, it is a technique that can be

implemented independently of program features to

prove. Selection of adequate MR is a critical aspect

in this technique, therefore, problem knowledge and

program structure must be considered.

We have proposed a diagram of a test framework

to apply MT to service compositions in WS-BPEL,

specifying steps and ways to design it.

Regarding future work, the proposed framework

must be formally speciﬁed, designed, and imple-

mented, developing every step until the system is

completed.

REFERENCES

ActiveVOS (2009). ActiveBPEL WS-BPEL Engine.

http://sourceforge.net/search/?q=ActiveBPEL.

Andrews, J. H., Briand, L. C., and Labiche, Y. (2005). Is

mutation an appropriate tool for testing experiments?

In Proceedings of the 27th International Conference

on Software Engineering (ICSE 2005), pages 402–

411. ACM Press.

Beizer, B. (1990). Software Testing Techniques, 2nd Edi-

tion. International Thomson Computer Press, 2 sub

edition.

Bozkurt, M., Harman, M., and Hassoun, Y. (2010). TR-10-

01: testing web services: A survey. Technical Report

TR-10-01, King’s College, London.

AN APPROACH TO METAMORPHIC TESTING FOR WS-BPEL COMPOSITIONS

141

Chan, W. K., Cheung, S., and Leung, K. (2006). Towards

a metamorphic testing methodology for service-

oriented software applications. In Quality Software,

2005.(QSIC 2005). Fifth International Conference on,

pages 470–476.

Chan, W. K., Cheung, S. C., and Leung, K. R. (2007).

A metamorphic testing approach for online testing of

service-oriented software applications. International

Journal of Web Services Research, 4(2):61–81.

Chen, T. Y. (1998). Metamorphic testing: A new approach

for generating next test cases. HKUSTCS98-01.

Chen, T. Y. (2010). Metamorphic testing: A simple ap-

proach to alleviate the oracle problem. In Proceedings

of the 5th IEEE International Symposium on Service

Oriented System Engineering. IEEE Computer Soci-

ety.

Chen, T. Y., Feng, J., and Tse, T. H. (2002). Metamorphic

testing of programs on partial differential equations:

A case study. In Proceedings of the 26th International

Computer Software and Applications Conference on

Prolonging Software Life: Development and Redevel-

opment, COMPSAC ’02, pages 327–333, Washing-

ton, DC, USA. IEEE Computer Society.

Chen, T. Y., Huang, D. H., Tse, T. H., and Zhou, Z. Q.

(2004). Case studies on the selection of useful re-

lations in metamorphic testing. In Proceedings of

the 4th Ibero-American Symposium on Software En-

gineering and Knowledge Engineering (JIISIC 2004),

pages 569–583.

Dom´ınguez-Jim´enez, J. J., Estero-Botaro, A., Garc´ıa-

Dom´ınguez, A., and Medina-Bulo, I. (2009). GAm-

era: An automatic mutant generation system for WS-

BPEL compositions. In ECOWS 2009: Seventh IEEE

European Conference on Web Services, pages 97–106,

Eindhoven, The Netherlands. IEEE Computer Soci-

ety.

Garc´ıa-Fanjul, J., Tuya, J., and de la Riva, C. (2007). Gen-

eraci´on sistem´atica de pruebas para composiciones de

servicios utilizando criterios de suﬁciencia basados en

transiciones. In JISBD 2007: Actas de las XII Jor-

nadas de Ingenier´ıa del Software y Bases de Datos.

Gotlieb, A. and Botella, B. (2003). Automated metamor-

phic testing. Computer Software and Applications

Conference, Annual International, 0:34–40.

IDC (2008). Research reports. http://www.idc.com.

Mayer, P. and L¨ubke, D. (2006). Towards a BPEL unit test-

ing framework. In TAV-WEB’06: Proceedings of the

2006 workshop on Testing, Analysis, and Veriﬁcation

of Web Services and Applications, pages 33–42, New

York, NY, USA. ACM.

Murphy, C., Kaiser, G., Hu, L., and Wu, L. (2008). Proper-

ties of machine learning applications for use in meta-

morphic testing. In Proc. of the 20th international

conference on software engineering and knowledge

engineering (SEKE), pages 867–872.

Murphy, C., Shen, K., and Kaiser, G. (2009). Using

JML runtime assertion checking to automate meta-

morphic testing in applications without test oracles.

In Software Testing Veriﬁcation and Validation, 2009.

ICST’09. International Conference on, pages 436–

445.

Myers, G.J., Sandler, C., Badgett, T., and Thomas, T. M.

(2004). The Art of Software Testing, 2nd ed. Wiley -

Interscience.

OASIS (2007). Web Services Business Process

Execution Language 2.0. http://docs.oasis-

open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html.

Organization for the Advancement of Structured

Information Standards.

Segura, S., Hierons, R. M., Benavides, D., and Ruiz-Cortes,

A. (2010). Automated test data generation on the anal-

yses of feature models: A metamorphic testing ap-

proach. Software Testing, Veriﬁcation, and Validation,

2008 International Conference on, 0:35–44.

UCASE Research Group (2010a). GAmera home site.

http://neptuno.uca.es/˜gamera.

UCASE Research Group (2010b). Takuan home

site. https://neptuno.uca.es/redmine/projects/takuan-

website.

Weyuker, E. (1982). On testing Non-Testable programs.

The Computer Journal, 25(4):465–470.

Yan, J., Li, Z., Yuan, Y., Sun, W., and Zhang, J. (2006).

BPEL4WS unit testing: Test case generation using a

concurrent path analysis approach. In ISSRE 2006:

17th International Symposium on Software Reliability

Engineering, pages 75–84, Raleigh, North Carolina,

USA. IEEE Computer Society.

Zhang, Z.-Y., Chan, W. K., Tse, T. H., and Hu, P.-F. (2009).

An experimental study to compare the use of meta-

morphic testing and assertion checking. Journal of

Software, 20(10):2637–2654.

Zheng, Y., Zhou, J., and Krause, P. (2007). An automatic

test case generation framework for web services. Jour-

nal of software, 2(3):64–77.

Zhou, Z. Q., Huang, D. H., Tse, T. H., Yang, Z., Huang,

H., and Chen, T. Y. (2004). Metamorphic testing and

its applications. In Proceedings of the 8th Interna-

tional Symposium on Future Software Technology (IS-

FST 2004). Software Engineers Association.

ICE-B 2011 - International Conference on e-Business

142