A GUI Modeling DSL for Pattern-Based GUI Testing

PARADIGM

Rodrigo M. L. M. Moreira

and Ana C. R. Paiva

INESC TEC and Department of Informatics Engineering

Faculty of Engineering of the University of Porto, Porto, Portugal

Keywords:

GUI Modeling, GUI Testing, Domain-Speciﬁc Languages, PARADIGM, Pattern-Based GUI Testing,

Graphical Modeling DSL.

Abstract:

Today’s software feature user interface (UI) patterns. Those patterns describe generic solutions for common

recurrent problems. However, to the best of our knowledge, there is no speciﬁc testing methodology that is

particularly suited for testing those patterns providing generic testing solutions that can be reused after minor

conﬁgurations in order to test slightly different implementations. Pattern-Based Graphical User Interface

Testing (PBGT) is a recent methodology that aims at systematizing and automating the GUI testing process,

by sampling the input space using “UI Test Patterns” that express generic solutions to test common recurrent

GUI’s behaviour. This paper describes the development process of PARADIGM, a domain speciﬁc language

(DSL) to be used in the context of PBGT and empirically evaluates PARADIGM to assess its diminished

modeling efforts, usefulness, graphical power, and acceptability.

1 INTRODUCTION

Nowadays the majority of software applications fea-

ture a Graphical User Interface (GUI). They are the

system’s entry point, upon which users communicate

with the underlying software. GUIs have become

quite popular and represent a big role on the success

of the software. GUIs stand as the ﬁrst contact be-

tween the system and its users. Users’ opinion is ex-

tremely important as it is their decision to either use

such software or not. If the user ﬁnds errors in the

GUI, he will lose trust on the software. Therefore, it

becomes important to test GUIs for their functional

correctness.

GUIs often feature UI Patterns (Moreira et al.,

2013)

. UI Patterns are recurring solutions to

solve common GUI design problems. Until quite re-

cently, no approach existed to test UI Patterns for

their functional correctness. Our prior work (Cunha

et al., 2010; Moreira et al., 2013) introduces a

new methodology called Pattern-Based GUI Testing

method (PBGT), that aims at systematizing and au-

tomating the GUI testing process. In addition, PBGT

develops the notion of UI Test Pattern, which provides

a reusable and conﬁgurable test strategy, in order to

test a GUI that was implemented using a set of UI

http://ui-patterns.com/, http://pttrns.com/

Patterns.

This paper describes PARADIGM, a DSL to be

used in the context of PBGT. It describes the activ-

ities followed during its development, according to

best practices and a set of guidelines, and presents an

empirical evaluation to assess its diminished model-

ing effort, usefulness, graphical power, and accept-

ability. Examples of the PBGT approach and the us-

age of PARADIGM to model real applications, can be

found in (Moreira et al., 2013).

Modeling GUIs for testing offers a number of

challenges. The need to express speciﬁc domain fea-

tures, along with the desire to express them using

paradigms familiar to the domain experts, leads to

the demand for domain speciﬁc languages. However,

DSLs are, in general, difﬁcult to design, implement

and maintain. In addition, developing DSLs from

scratch represent a laborous manual task. Moreover,

when/if a DSL grows in complexity and size, its de-

sign becomes more error-prone.

Alloy (Jackson, 2011) is an intuitive formal mod-

eling language, with a syntax that is simple easy to

read and write. It is supported by the Alloy Analyzer

tool, a friendly SAT (satisfability) based tool that en-

ables automatic model V&V (veriﬁcation and valida-

tion). In the context of our work, Alloy is used to sup-

port the development of a DSL from scratch. From

126

M. L. M. Moreira R. and C. R. Paiva A..

A GUI Modeling DSL for Pattern-Based GUI Testing - PARADIGM.

DOI: 10.5220/0004880601260135

In Proceedings of the 9th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE-2014), pages 126-135

ISBN: 978-989-758-030-7

 2014 SCITEPRESS (Science and Technology Publications, Lda.)

the Alloy model of the DSL, it is possible to gen-

erate instances (that should represent possible valid

GUI test models) and analyze those instances in or-

der to deﬁne and tune language constraints. This is

an iterative process that ends when the constraints

necessary to ensure the construction of well-formed

PARADIGM test models are found.

The main contributions of this current paper in-

clude:

– a detailed description concerning the process (ac-

cording to best practices) that was followed to cre-

ate a new DSL (PARADIGM) for the PBGT do-

main;

– a novel iterative approach using Alloy in DSL de-

velopment;

– an empirical case study to assess the effort re-

quired to model GUIs with PARADIGM;

– evaluation of PARADIGM usage at an industrial

environment.

2 BACKGROUND

This section presents existing GUI Modeling ap-

proaches and tools/frameworks that support the de-

velopment of graphical editors.

2.1 GUI Modeling

Spec# (Microsoft, 2013) is a model-based speciﬁca-

tion language developed by Microsoft Research. It

extends the existing object-oriented .NET program-

ming language C# with speciﬁcation constructs like

pre-conditions, post-conditions, invariants, and non-

null types. GUIs can be modelled with Spec# de-

scribing actions performed by the end-user and the

expected outcome of each user action (Paiva et al.,

2003). Hence, GUI Spec# models are the input to

Spec Explorer (Microsoft, 2012) with GUI Mapping

Tool (Paiva et al., 2005), which is a model-based test-

ing tool, that generates and executes test cases auto-

matically. However, the creation of GUI models with

Spec# is very demanding and time consuming.

The Visual Abstract Notation for GUI Model-

ing and Testing (VAN4GUIM) (Moreira and Paiva,

2008) is an approach that tries to diminish this ef-

fort, by hiding formalism details used in Model-Based

Testing (MBT) approaches. Furthermore, it aims

at promoting the use of MBT in industrial environ-

ments by providing a graphical front-end to assist

GUI modeling, and thus making the crafting of mod-

els more appealing to testers rather than the textual

notation. This visual notation is based on Canonical

Abstract Prototyping (Constantine, 2003) and Con-

curTaskTrees - CTT (Patern

o et al., 1997). It consists

of ﬁve UML proﬁles: Containers, who are respon-

sible to hold/group UI elements; User Actions, that

are tools to act upon containers; Hybrids that result in

combination between elements to model user actions

who act upon speciﬁc containers; CTT connectors to

describe relationship among elements and; Window

Manager to describe windows behavior. GUIs mod-

eled with VAN4GUIM are then translated automati-

cally to Spec#, according to a set of translation rules.

However, the model does not feature all GUIs behav-

ior. Additional behavior needs to be included manu-

ally in the generated Spec#. After the Spec# model

is completed it will be used as input to Spec Explorer

with GUI Mapping Tool.

Event Flow Graphs (EFG) are a popular approach

that allow to create a GUI model by capturing the ﬂow

of events, featuring all possible event interactions in

the UI (Memon et al., 2001). GUI models are gener-

ated via a tool called GUI Ripping Tool. However, to

the best of our knowledge, no tool exists to model an

EFG by hand.

2.2 Tools for Language Implementation

Numerous tools have the capability to generate graph-

ical editors from a metamodel of a DSL language.

Such tools differ on the: (i) implementation to attain

concrete graphical syntaxes; (ii) implementation com-

plexity; and (iii) maintenance efforts.

Microsoft Domain-Speciﬁc Language Tools

(Cook et al., 2007) allow to create a representation of

the model of a language, to specify their relations, to

describe the semantics of the language and to deﬁne

the DSL graphical representation. Furthermore,

Microsoft DSL Tools offer a solid and seamless

implementation, nonetheless depending on the im-

plementation goals, the fact of being platform reliant,

can be considered as a restriction.

The Eclipse platform contains a set of modeling

frameworks, such as Eclipse Modeling Framework

(EMF) (Steinberg et al., 2009), Graphical Editing

Framework (GEF) (Rubel et al., 2011) and Graphical

Modeling Framework (GMF) (Gronback and Boldt,

2013). EMF is a modeling framework that facilitates

building tools to support modeling languages. GEF

provides the graphical support needed for building a

model/language editor on the top of the EMF frame-

work. In addition, GMF is a more advanced and

generative framework for developing graphical edi-

tors for providing support for DSL development, by

leveraging EMF and GEF. These frameworks are pop-

ular, straightforward to use and maintain. Moreover,

AGUIModelingDSLforPattern-BasedGUITesting-PARADIGM

127

they have a strong community support.

StarUML (StarUML, 2005) is an open source

project having in mind ﬂexibility, extensibility, and

fast development. StarUML can be extended to pro-

vide further functionality over the tool by the devel-

opment of new modules. The tool acts as modeling

platform to provide functionality for various platform

technologies and external tools. This modeling soft-

ware offers great extensibility, but it requires a deep

knowledge over the core extension mechanisms of the

tool. Furthermore, this tool only runs on win32 sys-

tems and its development has stagnated.

Open ModelSphere (Grandite, 2008) is a platform

independent modeling tool. This tool is supported by

an experienced community and be freely modiﬁed as

desired. The downside is the complexity to cope with

the addition of new language elements. In addition, it

also requires a deep knowledge over the ModelSphere

layered model.

3 DSL IMPLEMENTATION

In general, DSLs are high-level languages exclusively

tailored to speciﬁc tasks. They are designed to make

their users effective in a speciﬁc domain. They are

important because they represent a more natural, ro-

bust, precise, and maintainable way of capturing the

essence of a given problem, rather than merely being

expressed in a general-purpose language.

The PBGT methodology requires a GUI model to

be crafted speciﬁcally for this domain. Thus, a lan-

guage is required to assist in the creation of mod-

els particularly tailored for PBGT. With the latter in

mind, several DSLs for GUI modeling have been eval-

uated, and have been described in the previous sec-

tion. However, to the best of our knowledge, there

is no DSL built from the right beginning of its devel-

opment with reusability concerns, providing UI Test

Patterns that can be adapted for testing different im-

plementations after a small conﬁguration step. This

language promotes reusability either by reusing exist-

ing elements or extending them to be reused by others.

In addition, our DSL allows building a model describ-

ing the test goals instead of describing the expected

behavior.

PARADIGM is a DSL that has been created, from

scratch, for PBGT. The main goals of this language

are:

– provide higher level of abstraction models;

– provide generic test strategies for testing GUIs;

– provide language ﬂexibility and promote reuse;

– provide extension mechanisms to cope with pos-

sible new UI Patterns trends;

– simple to use and maintain;

– and reduce the effort in building models.

PARADIGM development was based on a “tai-

lored DSL engineering process” (Strembeck and

Zdun, 2009), as well as guidance from Mar-

tin Fowler’s book “Domain Speciﬁc Languages”

(Fowler, 2010) and “Design Guidelines for Domain

Speciﬁc Languages” (Karsai et al., 2009). These ref-

erences deﬁne a set of best practices to follow during

the development of a DSL.

A DSL development is an iterative process. This

process is comprised by 5 major activities (Strem-

beck and Zdun, 2009): (1) Deﬁnition of the Language

Core Model with the language elements and their re-

lations; (2) Add constraints (if needed) to restrict the

language funcionalities (Language Restrictions); (3)

Specify the Syntax of the language to describe how

the elements are represented; (4) Describe the dy-

namic Behavior of the elements of the language, i.e.,

how such elements perform and interact; (5) Integrate

the DSL within a tool to support the construction of

models (Platform Integration).

3.1 Language Core Model

A language core model captures all relevant domain

abstractions and speciﬁes relations among them. Ab-

stractions refer to concepts within the context of

PBGT. In PARADIGM, these abstractions are behav-

ioral elements (UI Test Patterns), structural elements

and language connectors.

A DSL model written in UML (Figure 1) can be

deﬁned as a metamodel or a UML proﬁle. We opted

for a metamodel approach since (Brucker and Doser,

2007): (i) the deﬁned model does not need to be trans-

ferred to other domains; (ii) we want to have a more

abstract syntax; (iii) it provides easier semantic def-

inition; (iv) and the domain is well-deﬁned due to

a unique precise set of concepts (UI Test Patterns,

mainly).

3.1.1 Elements

An Element is an abstract entity that represents the

concepts within PBGT domain. A model written

in PARADIGM starts with the Init element and ﬁn-

ishes with the End. Both Init and End are special-

izations of the abstract entity Element. Models can

be structured in different levels of abstraction. For

instance, as models grow it is possible to use structur-

ing techniques, to handle different hierarchical levels,

ENASE2014-9thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

128

Figure 1: PARADIGM Language Metamodel.

so that a model A can “live” inside a model B. Struc-

tural Form elements were created speciﬁcally for this

purpose. A Form (Structural element) embodies a

model (or sub-model), with an Init and End elements.

Groups are also structural elements but they are used

to gather elements that may be executed in any or-

der. Behavioral elements represent the UI Test Pat-

terns that deﬁne strategies for testing the UI Patterns.

3.1.2 Connectors

ConcurTaskTrees (CTT) (Patern

o et al., 1997) are a

popular, expressive and powerful graphical notation

for task representation. It deﬁnes a set of temporal

operators to combine tasks. The connectors within

the PARADIGM language are inspired by CTT.

PARADIGM deﬁnes three connectors: “Se-

quence”; “SequenceWithDataPassing”; and “Se-

quenceWithMovedData”. The “Sequence” connector

indicates that the target element cannot start before

the source element has completed. The “Sequence-

WithDataPassing” connector has the same behavior

as “Sequence” and, additionally, indicates that the tar-

get element receives data from the source element.

“SequenceWithMovedData” has a similar meaning to

the “SequenceWithDataPassing” connector, however,

the source element transfers data to the target, so the

source loses the data that was transferred. In addi-

tion, there is another kind of relation among elements

– “Dependency” – indicating that the target element

depends on the properties of a set of source elements,

for instance, when it is the result of a calculation.

3.1.3 Language Constraints

The model of the PARADIGM’s language contains

additional constraints that cannot be expressed di-

rectly in the UML language model. During DSL de-

velopment some initials thoughts regarding language

constraints tend to evolve in time and need to be

enhanced as the language model increases and pro-

gresses. To assist in deﬁning the constraints to be

imposed on the language, we have decided to follow

an iterative development process by building the lan-

guage model in Alloy as will be explained in the se-

quel.

3.1.4 PARADIGM Model and Restrictions in

Alloy

Alloy is a lightweight formal language that supports

structural and behavioral modeling. It is suitable for

early stages of software development, which allows

discovering the correct software abstractions. Alloy

is based on relational logic that combines the quanti-

ﬁers of ﬁrst order logic with the operators of the rela-

tional calculus. In addition, Alloy allows expressing

complex structural and behavioral constraints. One

of the advantages of Alloy is its simple but powerful

notation.

An Alloy model contains a set of imports, signa-

tures, relations, facts, predicates, functions, assertions

and commands. A signature introduces a typed set of

atoms and may have ﬁelds. Facts are constraints on

relations that always hold. Predicates deﬁne reusable

constraints. Functions deﬁne reusable expressions.

Commands allow performing “check” and “run” anal-

ysis. Where “run” checks model consistency by look-

ing for valid instances and; “check” veriﬁes the asser-

tion looking for counter examples.

The PARADIGM UML metamodel (Figure 1)

was translated to Alloy in order to generate instances

(that should represent possible valid GUI test mod-

els) and, afterwards, analyze them to think about the

properties that such models should have and tune

the language constraints. This is an iterative process

that ends when the instances obtained from the Alloy

model correspond to valid well-formed PARADIGM

models. This is when the correct set of constraints is

found.

As such, the starting point of this process was to

deﬁne the structure of PARADIGM models as they

may be structured in different levels of abstraction.

From this Alloy model (Figure 2), it is possible to

generate instances (e.g., Figure 3) (with Alloy An-

alyzer Tool) for a scope of at most 11 elements of

each signature but exactly 2 Forms and 1 Group. We

have deﬁned an extra signature (comparing with the

AGUIModelingDSLforPattern-BasedGUITesting-PARADIGM

129

UML metamodel), Model, a specialization of Form

to describe the upper level of the overall model and

for simplifying the speciﬁcation. Figure 3 shows

a GUI test model with 2 hierarchical levels (Model

and Form). The Model has one Init (Init1), one End

(End1), one Behavior or UI Test Pattern (Behavior1),

a Form and a Group. The Form has one Init (Init0),

one End (End0) and a UI Test Pattern (Behavior0).

The Group has two UI Test Patterns (Behavior2 and

Behavior3). This instance allowed us to check if the

model speciﬁcation was correct (by featuring the de-

signed restrictions).

Figure 2: Alloy model featuring PARADIGM hierarchical

levels.

Figure 3: Instance of the Alloy model (from Figure 2) re-

garding PARADIGM hierarchical levels.

Afterwards, we evolved the model in order to in-

clude the description of the language connectors and

constraints (Figure 4) that ensure that each connec-

tor links only two elements that are within the same

Structural element (Model, Form and Group). Before

each statement there is a description of its meaning

starting with “– –”. An example of an instance gener-

ated from this model is illustrated in Figure 5.

The reﬁned language constraints for PARADIGM

are:

Figure 4: Alloy model featuring PARADIGM connectors.

Figure 5: Instance of the Alloy model (from Figure 4) re-

garding PARADIGM connectors.

LC1: A Connector cannot connect an element to

itself;

LC2: A Connector cannot have Init as destination

neither End as source;

LC3: An Init element cannot connect directly to

an End element;

LC4: Two elements cannot be connected more

than once by connectors of the same type;

LC5: Two Elements can only be connected if they

belong to the same Structural Element (Model; Form;

Group);

LC6: Elements inside a Form (but not inside

Groups of that Form) cannot be loose, i.e., for all el-

ements within a Form, there is at least one path from

the Init to the End that traverses that element.

3.2 Behavior Speciﬁcation

The DSL behavior speciﬁcation, also noted as dy-

namic semantics, stands as a part of the language

ENASE2014-9thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

130

model and deﬁnes the behavior concerning the us-

age of a given DSL language element. PARADIGM’s

behavioral semantics establishes the instructions on

how to use the language and deﬁnes the interactions

among language elements.

To some extent, the DSL language constraints al-

ready deﬁne the behavior of the PARADIGM lan-

guage. Nevertheless, these are not sufﬁcient to de-

ﬁne the behavioral varieties of the PARADIGM ele-

ments. So, it is important to also deﬁne the possi-

ble conﬁguration of the UI Test Patterns during the

modeling phase to allow testing generic behavior over

slightly different implementations and with different

input data.

3.2.1 UI Test Patterns

UI Patterns represent common functionalities that can

be implemented in different ways. Nevertheless, what

is important to notice is the common behavior that

is reﬂected in these generic implementations. A UI

Test Pattern aims at testing those UI Patterns accross

its several slightly different implementations. Mas-

ter/Detail, Find, Sort, Input and Call are UI Test Pat-

terns.

Deﬁnition 1. UI Test Pattern deﬁnes a test strategy

which is formally deﬁned by a set of tuples with the

form:

< Goal,V, A,C, P >, where

Goal is the id of the test; V is a set of pairs

{[variable, inputData]} relating test input data with

the variables involved in the test; A is the sequence

of actions to perform during test case execution; C

is the set of possible checks to perform during test

case execution, for example, “check if it remains in

the same page”; P is a Boolean expression (precondi-

tion) deﬁning the set of states in which it is possible to

execute the test. In another way, Goal is the “name”

of the test. A and the variables in V describe “what”

to do and “how” to execute the test. C describes the

ﬁnal purpose (or why) the test should be executed. P

deﬁnes when can be executed.

The Goal, variables in V, A and C are deﬁned

by the developer during the implementation phase of

the language. During the modeling phase, the tester

needs to conﬁgure each UI Test Pattern within the

model. He has to select the testing Goals (tests) and,

for each of those Goals, provide test input data, se-

lect the checks to perform, and deﬁne the precondi-

tion that describes the states where it is possible to

execute the corresponding actions. The tester can se-

lect the same Goal multiple times for a UI Test Pattern

providing different conﬁgurations.

Input UI Test Pattern. This UI Test Pattern

should be used to test the behavior of input ﬁelds for

valid and invalid input data.

– Goals: “Valid data” (INP VD) and “Invalid data”

(INP ID);

– Set of variables: {input};

– Checks to perform: {“message box”, “label”,

“error provider”};

– Sequence of actions: [provide input].

During conﬁguration, the user has to provide valid

input data for INP VD (and invalid input data for

INP ID), select the checks to perform and deﬁne the

precondition.

Master/Detail UI Test Pattern. This pattern

should be applied to test UI Patterns that feature two

areas, master and detail, related in such a way that,

when the master changes, the detail changes accord-

ingly. Examples of these UI Patterns can be found in

software applications such as iTunes

and Finder

– Goals: “Change master” (MD);

– Set of variables: {master, detail};

– Checks to perform: {“detail has value X”, “de-

tail does not have value X”, “detail is empty”};

– Sequence of actions: [select master].

During conﬁguration the user has to provide mas-

ter input data for MD, select the checks to perform

and deﬁne the precondition.

used to test an authentication. The goal is to check

if it is possible to authenticate with a valid user-

name/password and check if it is not possible to au-

thenticate with invalid usernames/passwords.

– Goals: “Valid login” (LG VAL) and “Invalid lo-

gin” (LG INV);

– Set of variables: {username, password};

– Checks to perform: {“change page”, “pop-up er-

ror”, “same page”};

– Sequence of actions: [provide username; provide

password; press submit].

During conﬁguration the user has to provide valid

username/password input data for LG VAL (and in-

valid username/password for LG INV), select the

checks to perform and deﬁne the precondition.

Find UI Test Pattern. This pattern should be used

when someone wants to test the result of a search that

shows up after a submit action. The goal is to verify

that the result of the search is as expected (it ﬁnds the

right set of values).

http://www.apple.com/itunes/

http://support.apple.com/kb/ht2470

AGUIModelingDSLforPattern-BasedGUITesting-PARADIGM

131

– Goals: “Value found” (FND VF) and “Value not

found” (FND NF);

– Set of variables: {v

,...,v

};

– Checks to perform: {“empty set”, “if it has X

elements”, “if it does not have element X”, “if the

result in line X is Y”};

– Sequence of actions: [provide v

,... , provide v

During conﬁguration, the user has to deﬁne the

value for N, provide input data for variables v

,...,v

select the check to perform and deﬁne the precondi-

tion.

Sort UI Test Pattern. The Sort UI Test Pattern

is used to check if the result (of a sort action) is or-

dered accordingly to the chosen sort criterion. The

idea is to test user interfaces that contain sortable

items/elements, such as, tables and lists.

– Goals: “ascending” (SRT ASC) and “descend-

ing” (SRT DESC);

– Set of variables: {v

,...,v

};

– Checks to perform: {“element from ﬁeld X in

position Y has value Z”};

– Sequence of actions: [provide v

,... , provide v

During conﬁguration, the user has to deﬁne the

value for N, provide input data for variables v

,...,v

select the check to perform and deﬁne the precondi-

tion.

Call UI Test Pattern. This UI Test Pattern is used

to check the functionality of the corresponding invo-

cation.

– Goals: “Action succeeded” (CL AS) and “Action

failed” (CL AF);

– Set of variables: { };

– Checks to perform: {“pop-up message”, “stay in

the same page”, “change to page X”};

– Sequence of actions: [press].

During conﬁguration, the user has only to select

the check to perform and deﬁne the precondition.

3.3 Syntax

The next step in the DSL development is the deﬁnition

of its syntax (Figure 6). Hence, we deﬁned the graph-

ical symbols for the language connectors and for the

language elements. The reasons for selecting a graph-

ical notation over a textual one, was based on the fol-

lowing aspects: (i) easier to understand; (ii) simpler

to navigate and; (iii) intuitive and faster learning for

end-users.

All PARADIGM’s elements and connectors are

deﬁned by: (i) an icon/ﬁgure to represent the element

graphically; (ii) a short label to name the element; (iii)

a number (between square brackets) to identify the el-

ement, according to its context; and (iv) a Boolean

value, also placed between square brackets, to indi-

cate if the element is optional.

Figure 6: PARADIGM’ Syntax.

3.4 Integration

Eclipse was the selected target platform for

PARADIGM. GMF was the selected framework

for the development of a speciﬁc tool, called

PARADIGM Modeling Environment (ME) (Monteiro

and Paiva, 2013), targeted to assist the construction

of PARADIGM models in the context of PBGT.

However, several tools for language implementation

have also been evaluated, but they failed in simplicity

in the development, maintenance and support aspects.

PARADIGM ME tool development incorporated

all the previous deﬁnitions of the language: ele-

ments and their properties, connectors, graphical

representation and constraints. These constraints

were implemented in OCL (Cabot and Gogolla,

2012)). In addition, new widgets were deﬁned just

for improving end-user interaction (menus, toolbars

and buttons). This tool provides support for modeling

GUIs using PARADIGM and further functionality:

UI Test Pattern conﬁguration; test cases generation

from models; test case execution and; test coverage

information. Since PARADIGM aims to be an

expandable language (to cope with latest UI trends),

new elements and connectors may be added. This

means the language can be expanded during imple-

mentation phase (development) and/or afterwards

(modeling phase). For instance, the tester is able to

create his own set of behavioral elements by reusing

and combining the existent ones within Forms. The

set of those Forms can be seen as a library of new

elements that can be reused within other model.

4 EMPIRICAL EVALUATION

To assess the effectiveness and real-world relevance

of our approach, we have conducted a set of case

ENASE2014-9thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

132

studies following guidelines from Runeson and H

ost

(Runeson and H

ost, 2009). The goal is to measure the

feasibility of the PBGT approach, starting from the

modeling phase until the generated test cases output.

More speciﬁcally, to assess the less efforts required

to model a GUI, to conﬁgure the UI Test Patterns and

the capability to model common UIs. Furthermore,

we also want to known the acceptance level of

PARADIGM, among GUI testers. Our evaluation

addresses the following research questions:

RQ1: What are the overall modeling efforts required

to craft a model with PARADIGM, when compared

with other Model-Based GUI Testing approaches?

RQ2: Are current language elements and connectors

adequate and effective to model UIs, with the goal to

ﬁnding errors?

RQ3: What is the graphical effectiveness and power

of PARADIGM?

4.1 Experimental Objects

italiancharts. This is a public site that reﬂects the

best selling music singles and albums in Italy. It can

be found at http://italiancharts.com (to be referred as

EO1).

The Address Book. The Address Book is an open

source address management system, that in short, al-

lows to create unlimited number of addresses and as-

sociated information (to be referred as EO2).

Professional Calendar/Agenda. This application is

written in C#, and provides a calendar view for cre-

ating appointments and day events (to be referred as

EO3).

zkCalendar. This tool features scheduling function-

ality, with daily, weekly and monthly views (to be re-

ferred as EO4).

Industrial Applications. Two industrial applications

(to be referred as IA1 and IA2) that due to conﬁden-

tiality terms cannot be revealed.

4.2 Experimental Setup

To address RQ1 we have selected three Model-

Based GUI Testing approaches, namely: Spec#,

VAN4GUIM, and PARADIGM. With these ap-

proaches, models have to be crafted manually.

The experiments were performed by two different

teams: one comprised by research students and the

other comprised by professionals working in a soft-

ware company (to test their two own internal devel-

oped applications). Elements from both teams had

not been in contact with PARADIGM developers nei-

ther had previous knowledge about PBGT, Spec# and

VAN4GUIM. We presented these 3 approaches to the

teams in two different sessions during which they had

the opportunity to clarify doubts in order to be pre-

pared to start the experiments. In order to address

RQ1 we measured the time (in minutes) required to

create and conﬁgure the models using the 3 different

approaches mentioned before. To address RQ2 and

RQ3 we gathered feedback from the teams to ver-

ify if they were able to model the proposed UIs and

to assess language acceptability, graphical effective-

ness and communicative aptitude. Thus, to evaluate

PARADIGM, we selected 8 dimensions, according

to Moody’s criteria (Moody, 2007), that in our per-

spective are the ones that can be applied to evaluate

PARADIGM. Then, once the teams ﬁnished the con-

struction of the models, we delivered a questionnaire,

featuring the selected dimensions. For each dimen-

sion, the teams were able to select only one value of

the following scale (in terms of perception): High – 3;

Medium – 2; Low – 1; Do not know/blank – 0. Fur-

ther, we also evaluated VAN4GUIM according to the

same criteria, in order to perform a comparision be-

tween PARADIGM and VAN4GUIM. Spec# was not

included in this comparision, since it is not a graphi-

cal language.

4.3 Findings

After completion of the case studies above, we ob-

tained the following results. Table 1 displays the time

measurements, in minutes, concerning the modeling

(displayed as M in the table cells) and conﬁguration

(displayed as C in the table cells) efforts that were

required for the experimental objects (EO). This table

registered the data to address RQ1.

The results regarding the evaluation of

PARADIGM and VAN4GUIM, according to

Moody’s quality criteria, are represented in Table

2. Each table cell shows the average value for each

dimension of each method. The total represents the

sum of the cells dividing by all dimensions (8). Table

2 displays the data to fulﬁll RQ2 and RQ3.

4.4 Discussion

By analyzing Table 1, we found that, on average,

it took 343 minutes to craft models with Spec#,

243 minutes with VAN4GUIM and 64 minutes

with PARADIGM. Regarding the conﬁguration ef-

fort, Spec# and VAN4GUIM are similar, i.e., teams

spent 48 and 45 minutes respectively. However, with

PARADIGM it took 31 minutes. Concerning these

results, it is important to mention that with Spec#

and VAN4GUIM the focus is directed towards mod-

AGUIModelingDSLforPattern-BasedGUITesting-PARADIGM

133

Table 1: Case studies results.

Method

Spec# VAN4GUIM PARADIGM

EO1

M:303

C:46

M:181

C:41

M:62

C:20

EO2

M:210

C:34

M:121

C:31

M:32

C:14

EO3

M:177

C:32

M:110

C:27

M:26

C:12

EO4

M:183

C:40

M:125

C:38

M:30

C:22

IA1

M:546

C:61

M:421

C:57

M:104

C:50

IA2

M:671

C:76

M:498

C:75

M:131

C:68

Table 2: PARADIGM and VAN4GUIM ﬁndings concern-

ing Moody’s quality criteria, with input collected from the

teams.

Quality Criteria

PARADIGM VAN4GUIM

Discriminability

2.2 2.0

Perceptual and

cognitive limits

2.8 2.5

Emphasis

2.9 2.8

Perceptual

directness

2.6 2.5

Structure

2.8 2.8

Identiﬁcation

2.7 2.5

Expressiveness

2.8 2.6

Simplicity

2.7 2.7

Total

21.5 / 8 ' 2.7 20.4 / 8 ' 2.6

eling user actions. With PARADIGM the modeling

is focused on testing goals. It is possible to conclude

that with PARADIGM the modeling efforts and also

the conﬁguration aspects are much lower rather than

Spec# and VAN4GUIM.

To assess the overall acceptability, the graphical

richness and communicative ability of PARADIGM,

we collected feedback through a questionnaire ful-

ﬁlled by the team members. The results are

shown in Table 2. We evaluated PARADIGM

against VAN4GUIM according to 8 dimensions from

Moody’s criteria. The values of each cell (from Ta-

ble 2) represent the teams perception and evaluation

of each dimension. The higher the proximity to-

wards value 3, the higher and better their percep-

tion is, in respect to the given dimension concern-

ing the graphical notation in question. For exam-

ple, the discriminability dimension consists on the

perception if language elements are easy to differ-

entiate. For this dimension, PARADIGM registered

the average of 2.2 and VAN4GUIM the average of

2.0. This result indicates that according to the scale,

PARADIGM has a better rating on this dimension.

Due to space restrictions, we cannot explain all di-

mensions. However, the meaning of each one can

be found in (Moody, 2007). As for the overall rat-

ings for all dimensions, PARADIGM registered a rat-

ing of 2.7 and VAN4GUIM obtained a rating of 2.6.

Thus, both notations appear to have similar accept-

ability, perceptiveness and graphical richness. How-

ever, PARADIGM registered the highest score (with a

slight difference) among the evaluation performed by

the teams. This fact indicates that PARADIGM lan-

guage elements are expressive, easy to identify, effec-

tive and graphically rich. Given the obtained positive

feedback, results also point that PARADIGM can be

adopted at industry level.

We are aware that more studies are needed which

can be seen as a threat to validity. Nevertheless, the

case studies presented in this paper point to the di-

minished effort in modeling GUIs with PARADIGM.

Furthermore, we also evaluated the DSL at industrial

level and the feedback was promising. Both modeling

and conﬁguration efforts for the models, depend on

the testing goals designed for each application. This

means that depending on the depth and detail of the

testing goals, the results could have been different.

5 CONCLUSIONS

This paper presented PARADIGM, a DSL to be used

in the context of PBGT, and the steps performed, ac-

cording to best practices, in the development of such

DSL. In addition, it also presents an empirical evalu-

ation of the PARADIGM comparing with other mod-

eling approaches (Spec# and VAN4GUIM) and an

evaluation at industrial level. The case studies re-

sults show that building models in PARADIGM re-

quires less effort when compared with Spec# and

VAN4GUIM. The results obtained from the industrial

case study are encouraging and indicate that the over-

all approach has, in fact, potential of being adopted

by the industry.

ACKNOWLEDGEMENTS

This work is ﬁnanced by the ERDF - European Re-

gional Development Fund through the COMPETE

Programme (operational programme for competi-

tiveness) and by National Funds through the FCT

- Fundac¸

ao para a Ci

encia e a Tecnologia (Por-

tuguese Foundation for Science and Technology)

within project FCOMP-01-0124-FEDER-020554.

ENASE2014-9thInternationalConferenceonEvaluationofNovelSoftwareApproachestoSoftwareEngineering

134

REFERENCES

Brucker, A. D. and Doser, J. (2007). Metamodel-based

UML Notations for Domain-speciﬁc Languages. In

Favre, J. M., Gasevic, D., L

ammel, R., and Winter, A.,

editors, 4th International Workshop on Software Lan-

guage Engineering (ATEM 2007). Nashville, USA.

Cabot, J. and Gogolla, M. (2012). Object constraint lan-

guage (OCL): a deﬁnitive guide. In Proceedings of the

12th international conference on Formal Methods for

the Design of Computer, Communication, and Soft-

ware Systems: formal methods for model-driven en-

gineering, SFM’12, pages 58–90, Berlin, Heidelberg.

Springer-Verlag.

Constantine, L. (2003). Canonical Abstract Prototypes for

Abstract Visual and Interaction Design. In Interac-

tive Systems. Design, Speciﬁcation, and Veriﬁcation,

volume 2844 of LNCS, pages 1–15. Springer-Verlag,

Berlin/Heidelberg.

Cook, S., Jones, G., Kent, S., and Wills, A. (2007). Domain-

Speciﬁc Development with Visual Studio DSL Tools.

Addison-Wesley Professional, ﬁrst edition.

Cunha, M., Paiva, A. C. R., Sereno Ferreira, H., and Abreu,

R. (2010). PETTool: A Pattern-Based GUI Testing

Tool. In 2nd International Conference on Software

Technology and Engineering (ICSTE’10), SFM’12,

pages 202–206.

Fowler, M. (2010). Domain Speciﬁc Languages. Addison-

Wesley Professional, 1st edition.

Grandite (2008). Open Modelsphere – Free

Modeling Software Open Source GPL.

http://www.modelsphere.org/. Accessed January,

2013.

Gronback, R. C. and Boldt, N. (2013). Graphical Modeling

Framework. http://www.eclipse.org/modeling/gmp.

Accessed April, 2013.

Jackson, D. (2011). Software Abstractions: Logic, Lan-

guage, and Analysis. MIT Press; 2nd Revised edition.

Karsai, G., Krahn, H., Pinkernell, C., Rumpe, B., Schneider,

M., and Vlkel, S. (2009). Design Guidelines for Do-

main Speciﬁc Languages. In Rossi, M., Sprinkle, J.,

Gray, J., and Tolvanen, J.-P., editors, Proceedings of

the 9th OOPSLA Workshop on Domain-Speciﬁc Mod-

eling (DSM’09), pages 7–13.

Memon, A. M., Soffa, M. L., and Pollack, M. E. (2001).

Coverage Criteria for GUI Testing. In In Proceedings

of the 8th European Software Engineering Conference

(ESEC) and 9th ACM SIGSOFT International Sym-

posium on the Foundations of Software Engineering

(FSE-9), pages 256–267. ACM Press.

Microsoft (2012). Model-based Testing

with SpecExplorer - Microsoft Re-

search. http://research.microsoft.com/en-

us/projects/specexplorer/. Accessed February,

2012.

Microsoft (2013). Spec# - Microsoft Re-

search. http://research.microsoft.com/en-

us/projects/specsharp/. Accessed January, 2013.

Monteiro, T. and Paiva, A. C. R. (2013). Pattern Based GUI

Testing Modeling Environment. In ICST Workshops,

pages 140–143.

Moody, D. (2007). What Makes a Good Diagram? Im-

proving the Cognitive Effectiveness of Diagrams in

IS Development. In Wojtkowski, W., Wojtkowski,

W., Zupancic, J., Magyar, G., and Knapp, G., editors,

Advances in Information Systems Development, pages

481–492. Springer US.

Moreira, R. M. L. M. and Paiva, A. C. R. (2008). Vi-

sual Abstract Notation for GUI Modelling and Test-

ing – VAN4GUIM. In Cordeiro, J., Shishkov,

B., Ranchordas, A., and Helfert, M., editors,

ICSOFT (SE/MUSE/GSDCA), pages 104–111. IN-

STICC Press.

Moreira, R. M. L. M., Paiva, A. C. R., and Memon, A.

(2013). A Pattern-Based Approach for GUI Modeling

and Testing. In Proceedings of the 24th International

Symposium on Software Reliability Engineering, IS-

SRE’13, Pasadena, CA, USA. IEEE Computer Soci-

ety.

Paiva, A., Faria, J. C. P., and Vidal, R. F. A. M. (2003).

Speciﬁcation-Based Testing of User Interfaces. In In-

teractive Systems. Design, Speciﬁcation, and Veriﬁ-

cation, 10th International Workshop, volume 2844 of

LNCS, pages 139–153. Springer.

Paiva, A. C., Faria, J. C., Tillmann, N., and Vidal, R. A.

(2005). A Model-to-Implementation Mapping Tool

for Automated Model-Based GUI Testing. In Lau,

K.-K. and Banach, R., editors, Formal Methods and

Software Engineering, volume 3785 of LNCS, pages

450–464. Springer Berlin Heidelberg.

Patern

o, F., Mancini, C., and Meniconi, S. (1997). Concur-

TaskTrees: A Diagrammatic Notation for Specifying

Task Models. In Proceedings of the IFIP TC13 In-

ternational Conference on Human-Computer Interac-

tion, INTERACT ’97, pages 362–369, London, UK,

UK. Chapman & Hall, Ltd.

Rubel, D., Wren, J., and Clayberg, E. (2011). The Eclipse

Graphical Editing Framework (GEF). Addison-

Wesley Professional, 1st edition.

Runeson, P. and H

ost, M. (2009). Guidelines for conduct-

ing and reporting case study research in software engi-

neering. Empirical Software Engineering, 14(2):131–

164.

StarUML (2005). Staruml – The Open Source UML/MDA

Platform. http://staruml.sourceforge.net/en/. Ac-

cessed January, 2013.

Steinberg, D., Budinsky, F., Paternostro, M., and Merks,

E. (2009). EMF: Eclipse Modeling Framework 2.0.

Addison-Wesley Professional, 2nd edition.

Strembeck, M. and Zdun, U. (2009). An approach for

the systematic development of domain-speciﬁc lan-

guages. Softw. Pract. Exper., 39(15):1253–1292.

AGUIModelingDSLforPattern-BasedGUITesting-PARADIGM

135