End to End Speciﬁcation based Test Generation of Web Applications

Khusbu Bubna

International Institute of Information Technology, Bangalore, India

Keywords:

Web Applications, Statechart, Model Checking, Selenium.

Abstract:

Using formal speciﬁcations to generate test cases presents great potential for automation in testing and en-

hancing the quality of test cases. However, an important challenge in this direction is the design speciﬁcations

which are at a more abstract level than the implementation, with many important implementation level de-

tails missing. To generate executable test cases, these implementation details must be included at some stage.

Though there has been a lot of work in test generation from speciﬁcation, all the existing methods suffer from

this problem: either the test cases are not executable, or the process involves a non-trivial, manual step of trans-

lating the abstract test cases to concrete test cases. In this work, we present an approach of speciﬁcation based

test generation for web applications that overcomes the above challenge: test generation is completely auto-

mated and the test cases are fully executable on a test execution framework (e.g. Selenium RC). Further, our

methodology allows generation of multiple sets of concrete test cases from the the same formal speciﬁcation.

This makes it possible to use the same abstract speciﬁcation to generate test cases for a number of versions

of the system. Throughout the paper, a case study of Learning Management System is used to illustrate our

approach.

1 INTRODUCTION

As web applications are becoming more and more

ubiquitous, modelling and testing web applications

correctly is becoming necessary. Formal speciﬁca-

tion languages are used to clarify customer’s require-

ments by removing ambiguity, inconsistency and in-

completeness in the software requirements and de-

sign process. Thus, they are helpful in reducing the

requirement errors in the early stages of the soft-

ware development life cycle. Formal speciﬁcations

are mostly used in critical systems where the cost of

failure is catastrophic. Formal speciﬁcations can be

used in several ways of design cycle, static veriﬁca-

tion and test generation being the most notable.

This paper presents the ACT (Abstract to

Concrete Tests) tool, which uses a novel approach

of generating concrete test cases from abstract test

cases by using data shared across different interac-

tions of the web application with the web server.

A formal speciﬁcation language ‘Statecharts’ (Harel,

1987) was used to model the navigation behaviour

aspect of web applications. Then abstract test cases

were generated from the State chart model in our ACT

tool. Abstract test cases cannot be directly executed

on the implementation since they are deﬁned using el-

ements and objects of the State chart model, and must

be ﬁrst transformed to concrete test cases. So ﬁnally,

concrete executable Selenium RC JUnit test cases are

generated from these abstract test cases in our ACT

tool which can be directly executed on the imple-

mentation. The remainder of the paper is organized

as follows. Section 2 gives a brief review of related

work, while Section 3 explains the proposed method-

ology used by our ACT tool. Section 4 illustrates the

State chart speciﬁcation for modeling the navigation

behaviour of web applications. Section 5 illustrates

the generation of test path from the State chart model.

Section 6 illustrates how abstract test cases are gener-

ated using Symbolic execution and SMT solver. Sec-

tion 7 illustrates the translation of abstract test cases

to concrete test cases. Section 8 concludes the paper

and Section 9 gives scope for future work.

2 RELATED WORK

A number of formal, informal and semi-formal mod-

els like automata (K. Homma and Togashi, 2009),

Statechart (M. Han, 2006), UML and OCL (Conallen,

1999)(Ricca and Tonella, 2001), UML based web

engineering, alloy, directed graph and control ﬂow

graphs, SDL, term rewriting systems, XML have been

296

Bubna, K.

End to End Speciﬁcation based Test Generation of Web Applications.

In Proceedings of the 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering (ENASE 2016), pages 296-302

ISBN: 978-989-758-189-2

proposed in various studies (M. H. Alalﬁ, 2009) for

modeling web applications. The authors in (Conallen,

1999)(Ricca and Tonella, 2001) have proposed UML

class diagram and the authors in (M. Han, 2006) have

proposed statechart for modeling web navigation. A

methodology for generation of concrete executable

tests from abstract test cases using a test automation

language, the Structured Test Automation Language

(STAL) was proposed in (N. Li, 2013). The authors in

(N. Li, 2013) have proposed a mapping between iden-

tiﬁable elements in the model to JUnit executable java

code. The author in (Torsel, 2013 ) have presented an

approach using domain speciﬁc languages to model

the navigation aspect of the web application and have

used a UI mapping XML ﬁle to generate concrete

test cases for Selenium and Canoo web test tools.

(A.A. Andrews, 2005) used ﬁnite state machines for

web application testing. In (A. Marcetto and Ricca,

2008), an approach that utilizes recorded user inter-

action data to construct a state machine model espe-

cially for testing AJAX functionality is presented. In-

put data is provided from the collected requests and

test oracles have to be created manually. The gen-

erated test sequences are translated into the test case

format of the Selenium test automation tool. The au-

thors in (H. S. Hong and Sokolsky, 2001) have used

model checking to generate test cases for control ﬂow

and data ﬂow coverage criteria. (Harel, 1987)

3 SYSTEM ARCHITECTURE

Figure 1 gives an overview of the approach that the

ACT (Abstract to Concrete test) tool uses for gener-

ating concrete Selenium RC JUnit test cases from the

formal State chart web navigation model. The nav-

igation behaviour of our case study web application

is modeled using the formal speciﬁcation language

’State chart’. The front end which generates test paths

from the State chart model can be a test path generat-

ing algorithm like model checking or graph coverage

algorithm. Abstract test cases are generated from the

test paths with the help of the State chart speciﬁcation.

Then the abstract test cases generated are converted to

Figure 1: Proposed Test Generation Method used by our

ACT tool.

Selenium RC JUnit concrete test cases using the Map-

pings XML ﬁle. Each of these steps are explained in

elaborate details in the remaining sections.

4 MODELLING WEB

NAVIGATION USING STATE

CHARTS

From among the various ﬁnite state based formal

speciﬁcation languages, we have used Statecharts

(Harel, 1987) for modeling the navigation behaviour

of web applications.

Figure 2 shows the State chart speciﬁcation of our

case study, the Learning Management System (LMS)

used within our institute. In the Statechart speciﬁca-

tion shown in Figure 2, Inactive state denotes that the

web application has not yet started operating. The

composite state Active denotes that the web applica-

tion is in an operating state. Inside Active state, each

web page was modelled as a separate state. When the

user navigates from one webpage to another, there is

a transition between the corresponding states. LMS

agement System. Here we have used the mathemat-

ical notations of sets and ﬁrst order predicate logic

constructs in guards and actions of the transitions in

State chart model.

In a web application, the value which is entered

by the user in one interaction of the web application

with the web server is often used back in another in-

teraction of the web application with the web server.

For example, as shown in the State chart model of

Figure 2, admin can register a student in the LMS

System by entering values in the username and pass-

word input ﬁelds in the Add Student User Page. A

necessary requirement for a student to login in the

LMS Login Page is that the student must be regis-

tered beforehand and so must have a valid student

username and password. Thus there is a data ﬂow

between Add Student User Page page and LMS Lo-

gin Page. The variables valid_student_usernames and

valid_student_passwords are deﬁned in the transition

from state Add Student User Page to Successfully Stu-

dent Added Page and are used in the transition from

state LMS Login Page to My Courses web page.

The variables valid_student_usernames,

valid_student_passwords, valid_teacher_usernames

and valid_teacher_passwords are shared between

different interactions of the web application with the

web server and across different users using the web

application. When the web application has started

operating, these variables are initialized to the null

End to End Speciﬁcation based Test Generation of Web Applications

297

!!!

"#$%&'()!

"#'&'$*'+)!,!

valid_student_usernames=null/^/

valid_student_passwords=null/^/

valid_teacher_usernames=null/^/

valid_teacher_passwords=null/^/

valid_admin_username=``admin’’/^/

valid_admin_password=``admin’’/

-./&012#!

345!316'#!7$6)!

48!91/:-)-!!

48!91/:-)-!

;0<'#!=1<)!!

7$6)!

;00!>-):-!

Add/

Users.clicked/

Add/

Student.clicked/

;00!5&/0)#&!

!>-):!7$6)!

;00!?)$%.):-!!

>-):!7$6)!

Add/Teacher.clicked/

5/%%)--@/**8!

5&/0)#&!;00)0!

7$6)!

Add.clicked/

A!/-):#$<)B!

∉!valid_student_usernames!∧

C$--21:0B!

∉!valid_student_passwords!D!,!

valid_student_usernames/:=/

valid_student_usernames/

∪/username///∧/

valid_student_passwords/:=/

valid_student_passwords/

∪/password//

5/%%)--@/**8!

?)$%.):!;00)0!!

7$6)!

E::1:!7$6)!

Add.clicked/

A!/-):#$<)B!

∈!

valid_student_usernames!!

∧!

C$--21:0B!

∈!

valid_student_passwords!D!

Add.clicked/

A!/-):#$<)B!

∈!!

valid_teacher_usernames/

∧!

C$--21:0B!

∈!

valid_teacher_passwords!D!

Add.clicked/

A!/-):#$<)B!∉ !"#$%_!"#$ℎ!" _!"#$%&'#"!∧!

C$--21:0B!∉ !"#$%_!"#$ℎ!"_!"##$%&'#!D!,!

valid_teacher_usernames/:=/valid_teacher_usernames///

∪ !"#$%&'#!! ∧/

valid_teacher_passwords/:=/valid_teacher_passwords/

∪/password!

A!/-):#$<)F!

∈!valid_student_usernames!!∧!

C$--21:0F!∈!valid_student_passwords!!D!

A!/-):#$<)!G!valid_admin_username!/

∧!

!!!!C$--21:0!G!valid_admin_password!D!

A!/-):#$<)H!

∈!!valid_teacher_usernames!/∧/

!!!C$--21:0H!!

∈!!valid_teacher_passwords!D!

95I5EIFJK!L)-'6#M$#0M;#$*8-'-!!

MN@M5$@)&8I9:'&'%$*M58-&)<-!

5EIFHKM51@&2$:)!

M?)-&'#6!

Design/and/Analysis/

of/

Safety/Critical/

Systems.clicked/

O)2-!P1:/<!

Q)#):$*!L'-%/--'1#!

P1:/<!

Q:$0)-!7$6)!

5*'0)-!P1*0):!

C$6)!

Grades.clicked/

News/

Forum.clicked/

Software/

Testing.clicked/

General/

Discussion/

Forum.clicked/

Slides/

Folder.clicked/

?)-&!L:'()#!!

L)()*1C)<#&!

;--'6#<)#&!7$6)!

TDD/

Assignment.clicked/

Logout.clicked/

;%&'()!

A!/-):#$<)F!

∉!valid_student_usernames/⋁!!!

!!C$--21:0F!!

∉!valid_student_passwords!!!D!

Logout.clicked/

A!/-):#$<)!

∉!valid_teacher_usernames/⋁!!!

!!C$--21:0!!

∉!valid_teacher_passwords!!!D!

Figure 2: Formal Model showing Web Navigation of Learning Management System using UML State chart model.

set. For example as shown in the State chart diagram

of Figure 2, variables valid_student_usernames,

valid_student_passwords, valid_teacher_usernames

and valid_teacher_passwords are initilized to null set

in the transition from state Inactive to state Active.

On a successful registration of either a teacher or a

student, the values of these variables are updated and

when either a student or a teacher logs in the LMS

to check if a student or teacher is already registered

or not.

Our ACT tool, as illustrated in the remaining sec-

tions can store and share data across different interac-

tions of the web application with the web server and

can generate Selenium RC JUnit concrete test cases

utilizing this data.

5 GENERATION OF TEST PATHS

FROM STATE CHART MODEL

USING MODEL CHECKING

The front end step in the ACT tool which generates

test paths can be any test path generating algorithm

like graph coverage algorithms. But model checking

which is a formal veriﬁcation technique can also be

used to generate test paths by formulating a tempo-

ral logic speciﬁcation as a trap property to be veriﬁed.

The straight forward way to represent a State chart

as a transition system is to ﬂatten its hierarchy. In

our approach, the hierarchical State chart navigation

model was ﬁrst ﬂattened and then transformed into

an SMV program. The trap properties for navigation

were written in CTL formulas and then the Symbolic

Model Veriﬁer (NuSMV) tool is executed which gen-

erates the counter examples. For example, the CTL

Trap Speciﬁcation Property for specifying that state

SE-230_Software_Testing is reachable from the ini-

tial state is:

CTL Trap Speciﬁcation Property:

! EF(state=SE-230_Software_Testing)

This trap property will generate a counter example

which is our test path. For this counter example there

will be two paths. One from LMS Login Page to My

Courses Page to SE-230 Software Testing Page. And

the second test path will be the loop from LMS Login

Page to Successfully Student Added Page then to My

Course Page to SE-230 Software Testing page. Here

ENASE 2016 - 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering

298

we will restrict the length of the loop in the counter

example by using bounded model checking and set

the length of the counter example to a ﬁxed size. The

CTL Trap properties for generating test paths are writ-

ten for various requirements of the web application,

like the top page of a web application should be reach-

able from all the pages of the web application. In ad-

dition, we also wrote CTL trap properties for node

coverage criterion.

6 ABSTRACT TEST CASES

GENERATION USING

SYMBOLIC EXECUTION AND

SMT SOLVER

After the test paths are generated from the State chart

model, symbolic execution is carried out to gener-

ate input values to guide the execution along the test

paths. The path predicate corresponding to the test

path is computed. This, in turn, is given to an SMT

solver to generate concrete input values. These val-

ues are used to generate the abstract test cases. The

algorithm used to generate the abstract test cases us-

ing symbolic execution is shown in the block diagram

in Figure 3.

6.1 Path Predicate Formation &

Computation of Concrete Values

This section illustrates the method of forma-

tion of path predicates and the generation

of concrete values using the example of the

counter example generated from the CTL Trap

Property !EF(state=SE-230_Software_Testing).

The variables valid_student_usernames and

valid_student_passwords in the State chart model

in Figure 2 are internal program variables and are

internally maintained as sets. These variables are

initialized to null in the transition from state Inactive

to composite state Active as shown in Figure 2. The

path predicate formed for the transition from Add

Student User Page to Successfully Student Added

Page is

(username1 6= ””) ∧ (password1 6= ””) ∧

(username1 /∈ ) ∧ (password1 /∈ ).

Sets valid_student_usernames and

valid_student_passwords are consequently up-

dated to username1 and password1. So the path

predicate for the transition from state LMS Login

Page to My Courses is

((username1 6= ”” ∧ (password1 6= ””) ∧

(username1 /∈ φ) ∧ (password1 /∈ φ)∧

ABS T R ACT TEST

CA S E GENERATION

Path

Spec

Symbolic

Execution

Path

Predicate

Extraction

SMT

Solver

Form Test

Case

Abstract

Test

Case

Symbolic

Execution

Trace

Path

Predicate

Concrete

Values

Figure 3: Abstract test case generation using Symbolic ex-

ecution.

(username2 ∈ username1) ∧ (password2 ∈

password1))

So the ﬁnal path predicate for the counter example

! EF(state=SE-230_Software_Testing ) is:

((username1 6= ”” ∧ (password1 6= ””) ∧

(username1 /∈ φ) ∧ (password1 /∈ φ) ∧ (username2 ∈

username1)∧ (password2 ∈ password1))

Here username1 and password1 are the symbolic

variables corresponding to the Add_Student_User

page, and username2 and password2 are the symbolic

variables corresponding to the LMS Login page. An

SMT solver like CVC3 (Key, ) would determine if

the path predicate is satisﬁable, and if yes, then it will

generate satisfying values for the symbolic variables

occurring in the formula and hence the concrete input

values would be generated.

Suppose, if a counter example is generated for the

path from state Inactive to state Error Page through

states LMS Login, Admin Home Page, Add Users

and Add Student User Page, then the path predicate

formed will be:

(username1 ∈ φ) ∧ (password1 ∈ φ)

which will be a contradiction i.e. unsatisﬁable

predicate. Hence, this path will be rejected as infeasi-

ble.

6.2 Abstract Test Cases Generation

Finally the abstract test cases are generated by plug-

ging in concrete input values which are computed in

the symbolic execution stage at appropriate points in

the counter examples. The phases in the counter ex-

amples which have corresponding GUI elements in

system implementation are extracted in sequence and

their user input ﬁelds are replaced by concrete values

computed in symbolic execution stage.

End to End Speciﬁcation based Test Generation of Web Applications

299

Table 1: Some CTL Trap Temporal Logic Properties that were used for generating test paths for LMS system in NuSMV tool.

Test Criterion

CTL Trap Property Description

Node coverage

! EF (state=SE-230_Software_Testing)

A CTL Trap property for node coverage

for state SE-230 Software Testing Page.

The top page is

reachable from all the

pages

AG((state=SE-230_Software_Testing) →

EF(state=LMS_Login_Page))

LMS Login Page is reachable from

SE-230 Software Testing Page.

! AG((state=Add_Student_User_Page) →

EF(state=LMS_Login_Page))

LMS Login Page is reachable from Add

Student User Page.

Every page is reachable

from the top page

! AG ((state=LMS_Login_Page) → EF

(state=SE-230_Software_Testing))

SE-230 Software Testing Page is

reachable from LMS Login Page page.

! AG ((state=LMS_Login_Page) → EF

(state=Add_Student_User_Page))

Add Student User Page is reachable from

LMS Login Page.

username=“admin”

password=“admin”

Add Users.clicked

Add Student.clicked

username1=“MS2013009”

password1=“abc”

Add.clicked

Logout.clicked

username2=“MS2013009”

password2=“abc”

Software Testing.clicked.

Figure 4: An example of an abstract test case generated for

the counter example generated from the CTL Trap Tempo-

ral Logic Property !EF(state=SE-230_Software_Testing).

7 TRANSLATION OF ABSTRACT

TEST CASES TO CONCRETE

TEST CASES

The last step is deriving concrete executable test cases

from these abstract test cases so these concrete test

cases can communicate directly with the system under

test. The abstract test cases generated from the State

chart model are on the same level of abstraction as the

model since the State chart model they are generated

from contains only partial information of the imple-

mentation under test. In Figure 5, even through the

input values are concrete, the test case is still abstract

because ﬁrstly variables like username1, password1,

etc. needs to be mapped to the implementation enti-

ties . And, secondly many of the inputs that may be

needed may not be there in the original speciﬁcation

and are only available after the exact implementation

is ﬁnished.

For generating concrete test cases from the ab-

stract test cases generated in the previous step, the

ACT tool uses a mapping between phrases used in

the State chart speciﬁcation model to Selenium Re-

mote Control JUnit test code which helps to translate

an abstract test to concrete Selenium RC JUnit test.

Figure 8 shows a part of XML ﬁle that was created to

give a mapping between the phrases in the State chart

model of Figure 2 to Selenium RC JUnit java code for

our case study of Learning Management System. The

mappings XML ﬁle is created manually.

Since, the State chart model contained only partial

information of the ﬁnal implementation. So, now the

additional information that were abstracted out from

the State chart model has to be added in the Map-

pings XML ﬁle in order to generate the concrete Se-

lenium RC Junit test cases. From the source code

implementation of the case study of HMS, we found

that in the Add Student User page there were addi-

tional input ﬁelds like Name and Department which

were abstracted out in the State chart speciﬁcation of

Figure 2. So these ﬁelds were also included in the

Mappings XML ﬁle. Using the generated abstract test

cases from the previous step and the mappings XML

ﬁle created manually and fed as input to the ACT tool,

a set of concrete Selenium RC Junit test cases is gen-

erated in this step. Figure 5 shows one of the concrete

Selenium JUnit test case generated from our ACT tool

for the case study of Learning Management System.

8 EXPERIMENTAL RESULTS

For two other web based enterprise applications de-

veloped within our institute, a Hospital Management

System (HMS) and a Student Information System

(SIS), we modeled our web applications using the

proposed method.

Table 2 shows the lines of code (LOC) of the two

systems, the number of nodes in the ﬂattened State

chart (N

f lat

), the number of mappings and the number

of abstract test cases( or counter examples) used for

the various test coverage criterion. A total of 14 ab-

stract test cases were used for HMS system and a total

of 53 abstract test cases were used for SIS System for

ENASE 2016 - 11th International Conference on Evaluation of Novel Software Approaches to Software Engineering

300

Table 2: Results of test generation on two other case studies Hospital Management System and Student Information System.

Web Applications N

f lat

LOC No. of Mappings

No. of abstract tests

(counter examples)

Hospital Management System

(HMS)

6 133 20

Node coverage 6

Top page is reachable from all the pages 4

Every page is reachable from the top page 4

Total 14

Student Information System

(SIS)

19 15,770 56

Node coverage 19

Top page is reachable from all the pages 17

Every page is reachable from the top page 17

Total 53

<phrase>state</phrase>

<value>Add_Student_User</value>

<code>

selenium=new DefaultSelenium(“localhost”,4444,”ﬁrefox/”

+“Applications/Firefox.app/Contents/MacOS/”

+“ﬁrefox-bin”,“https://lms.iiitb.ac.in/moodle”);”

selenium.start();

selenium.open(“/course/view.php?id=19”);

</code>

</mapping>

<phrase>username1</phrase>

<range>non blank</range>

<code>selenium.type (“css=input[id=’username’,]”,u1);</code>

</mapping>

<code> selenium.type(“css=input[id=’name’]”,n); </code>

</mapping>

<phrase>Department</phrase>

<code> selenium.type(“css=input[id=’department’]”,dp);</code>

</mapping>

</mappings>

Figure 5: A part of Mappings XML ﬁle showing mappings

between phrases used in State chart speciﬁcation model of

Figure 2 to their corresponding Selenium RC JUnit code.

various test criterion. Currently, we have modelled

the two case studies of HMS System and SIS System

but we are currently running our ACT tool and try-

ing to calculate the number of concrete test cases that

would be generated.

9 CONCLUSION

Test cases generated from formal speciﬁcations are

often at an abstract level. They cannot be executed

directly using a test automation tool, e.g. Selenium.

The existing methods have tried to deal with this prob-

lem using a mapping approach. However, this ap-

public void test1()

{

selenium.start();

selenium.open("https://lms.iiitb.ac.in/moodle/login/index.php");

selenium.type("css=input[id=’username’]", "admin");

selenium.type("css=input[id=’password’]", "admin");