IMPROVING SOFTWARE TEST STRATEGY WITH A METHOD

TO SPECIFY TEST CASES (MSTC)

Edumilis Méndez, María Pérez and Luis E. Mendoza

Processes and Systems Department – LISI

Universidad Simón Bolívar

Caracas – Venezuela

Keywords: Software testing, Test cases, Method, Test strategy, Use cases, Software test.

Abstract: An interesting difference between tests and other disciplines of the software development process is that

they constitute a task that essentially identifies and evidences the weaknesses of the software product. Four

relevant elements are considered when defining tests namely, reliability, cost, time and quality. Time and

cost shall increase to the extent reliable tests and quality software are desired, but what does it take to make

actors understand that tests should be seen as a security network? If quality is not there before starting the

tests, it will not be there upon their completion. Accordingly, how can we lay out a trace between tests and

functional and non-functional requirements of the software system? This Article is aimed at proposing a

method that allows for specifying test cases based on use cases, by incorporating elements to verify and

validate traceability among requirements management, analysis & design, and tests. This initiative

originated as a response to the request of a software developing company of the Venezuelan public sector.

1 INTRODUCTION

The objective of the test discipline is to assess

product quality throughout its life cycle based on a

set of best practices (Leffingwell and Widrig, 2006)

that include the following: (a) verification of the

software product’s proper operation, and (b)

verification of requirements’ proper implementation.

Grimán et al. (2003) indicate that this discipline is

not usually implemented in an organized and

systematic manner. Additionally, according to

Kruchten (2000), Pfleeger (1998), Pressman, (2002),

and Sommerville (2000), the test execution process

must be considered throughout the project life cycle

to ensure a high quality product. Success of this

process depends on the adoption of an adequate

testing strategy. A software testing strategy

comprises a group of activities that describes the

steps to be taken in a test process, considering the

amount of efforts and resources required for

achieving proper software construction (Pressman,

2002).

But, from the perspective of a company, which

strategies can be used? Which methodology or

method should be adopted to determine the

traceability between tests and requirements? Which

methodology or method ensures enhanced

verification and validation activities? Which

strategy guarantees the delivery of a quality

software product?

In this regard, we proposes a method that allows

specifying test cases (TCs) based on use cases

(UCs), as a starting point for the standardization and

traceability of the software development process,

thus obtaining highly profitable quality products.

2 RELATED WORK

The IEEE Standard for Software Test

Documentation (IEEE829-98) provides a good

description of test documents and their relationship

with one another and with the testing process. Test

documents may include, among others, TC

Specification. (SWEBOK, 2004).

Three key aspects (Utting and Legeard, 2007)

are considered for functional testing: design of the

test case, application of the test and analysis of

results, and verification of how the test fulfills the

requirements.

Perry (2006) introduces a complete guide for

effective testing process, including TCs, and it

proposes a TC template containing certain aspects

159

Méndez E., Pérez M. and E. Mendoza L. (2008).

IMPROVING SOFTWARE TEST STRATEGY WITH A METHOD TO SPECIFY TEST CASES (MSTC).

In Proceedings of the Tenth International Conference on Enterprise Information Systems - DISI, pages 159-164

DOI: 10.5220/0001686501590164

 SciTePress

considered for our method, such as the use of IDs

for TCs and UCs. Likewise, Lewis (2000) proposes

a template for TCs that includes conditions,

environment, version and system.

In recent years, a Model-based Testing (TDD)

approach was proposed, which provided for the

automation of the design of black-box tests.

SWEBOK (2004) defines black-box test where test

cases rely solely on the input/output behavior.

Some authors like Pinkster et al. (2006), state

that subsequent improvement in testing quality is

more than likely provided that requirements are

considered as the testing basis; this is known as

requirement-based testing.

Likewise, UML Testing Profile (2005)

introduces the integration of concepts, such as test

control, test group, and TC into the TC concept,

which can be decomposed into several lower-level

test cases and permits the easy reuse of TC

definitions in various hierarchies.

Some benefits of the requirements/test matrix

(Lewis, 2000) include: correlation of tests and

scripts with requirements, facilitation of review

status, and provision of a traceability mechanism

throughout the development cycle that includes test

design and execution.

In contrast to prior initiatives, our method

includes all the aforementioned ideas for the

purpose of obtaining a method that supports

elements comprised in a test strategy.

3 TEST CASE

A TC is a specification -usually formal- of a set of

test inputs, execution conditions and expected

outputs identified for the purpose of assessing the

particular aspects of a testing element (Leffingwell

and Widrig, 2006): (a) TCs reflect traceability with

UCs (functionality), (b) TCs include the

complementary specifications of the requirements,

and (c) TCs provide the system’s design

specifications.

All these elements ensure the compatibility of

test procedures with user/consumer requirements. In

practice, it is assumed that a UC itself is a TC and

that the project team works on the UCs without

planning the TCs in advance. As they test UCs, they

intuitively assume the test data and procedures

without making the need of documentation, which is

rather a mistake, since TCs expand or enhance the

information included in UCs. For instance, for UCs,

the values or conditions of tests are not specified.

TCs are essential for all testing activities

(Leffingwell and Widrig, 2006) due to the

following:

• They constitute the basis for the design and

execution of test procedures.

• Tests’ depth is proportional to the number of

TCs.

• Design and development, as well as required

resources, are governed by the required TCs.

If TCs are incorrect, the system quality will not

be reliable.

The method proposed herein states that testing

procedures are comprised of steps, conditions,

values, and expected/obtained results. Moreover, the

testing procedure may be automated through test

scripts. All the aforementioned concepts allow for

visualizing the test scope: What will be tested?

How, who, when, and what for? Once all TCs are

executed, the results should be fully disclosed for

the purpose of determining whether the acceptance

criteria defined by the user were satisfied upon

system validation.

In the following section you will find more

details of the proposed method.

4 METHOD TO SPECIFY TEST

CASES (MSTC)

The proposed method consists in creating a set of

TCs from a UC, since it is assumed that software

behavior must be tested based on requests or

requirements. Moving from a UC to a corresponding

set of TCs implies a reasonably wide and nontrivial

process. Leffingwell and Widrig (2006) describe

four (4) steps for achieving this process. Such steps

indicate what it is to be done, but they do not

explain in detail how to do it. Certain aspects that

were not expressed in writing were gathered and

proposed through the MSTC, based on bibliographic

review and our experience. Considering those steps

proposed by Leffingwell and Widrig (2006), we

intend to provide a method for specifying a set of

TCs from a UC.

The MSTC contribution consists in the

incorporation of tests’ traceability elements as to

the entire development process and enhancement of

the testing strategy while regulating this process.

The development process is then structured in

phases, activities, roles or individuals involved and

artifacts generated. Likewise, it helps documenting

ideas that were issued prior to tests, and explaining

how TCs were generated. This is useful for verify

ICEIS 2008 - International Conference on Enterprise Information Systems

160

tests traceability with respect to previous phases,

and requirement, analysis and design disciplines,

and it guarantees the organization’s knowledge

management regarding quality assurance.

This method includes the 4 roles proposed within

the test discipline: test manager, test designer, test

analyst and tester. Each phase is described in the

following paragraphs, but due to space limitations,

the researchers’ contribution will be highlighted in

italics.

It should be mentioned that MSTC is activated

by the test analyst once UCs narratives are verified

and upon system functionalities’ approval by the

stakeholders.

4.1 Phase 1: Scenarios’ Identification

Activities in this phase to be conducted by the test

analyst are as follows:

1. Scenarios are identified based on the UC

narratives and considering specific scenarios for

each UC. The regular flow, each alternate flow or a

combination of both represents a scenario

susceptible of being executed and tested.

Consequently, the first scenario will always evoke

the regular flow of that particular UC. The relations

between the UCs and the scenarios may be one-to-

many.

2. Graphical representation of the sequence of

events for each UC: As shown in Figure 1, this

allows for abstracting events taking place in a UC,

i.e. the regular or basic flow and alternate flows, and

it helps visualizing the potential combinations that

would represent a scenario, since it determines the

point at which the basic flow occurs and also what

happens upon alternate flow activation. Then, the

UC is completed or returned to the basic flow.

3. Verification that all alternate flows, including

their completion or return actions, were graphically

represented.

4. Illustration (as seen in Table 1) of all scenarios

associated to a UC in Figure 1.

5. Identification of all UC’s scenarios, indicating

regular and/or alternate flow(s) comprised within

the UC. Table 2 represents the first of 3 devices

generated in the MSTC: Table: Scenarios per UC.

In this table, we may observe that the ID

scenario is entered for the purpose of establishing

the tests’ traceability element, thus facilitating the

verification and approval of the tests and related

UCs. As can be seen in Table 2, IDs may include the

number of UCs and scenarios.

Figure 1: Flow visualization in a UC (Leffingwell and

Widrig, 2006).

Table 1: Scenarios per UC (Leffingwell and Widrig,

2006).

Number of

scenario

Originary flow Alternate flow Alternate next Alternate next

Scenario 1 Basic flow

Scenario 2 Basic flow Alternate flow 1

Scenario 3 Basic flow Alternate flow 1 Alternate flow 2

Scenario 4 Basic flow Alternate flow 3

Scenario 5 Basic flow Alternate flow 3 Alternate flow 1

Scenario 6 Basic flow Alternate flow 3 Alternate flow 1 Alternate flow 2

Scenario 7 Basic flow Alternate flow 4

Scenario 8 Basic flow Alternate flow 3 Alternate flow 4

Number of

scenario

Originary flow Alternate flow Alternate next Alternate next

Scenario 1 Basic flow

Scenario 2 Basic flow Alternate flow 1

Scenario 3 Basic flow Alternate flow 1 Alternate flow 2

Scenario 4 Basic flow Alternate flow 3

Scenario 5 Basic flow Alternate flow 3 Alternate flow 1

Scenario 6 Basic flow Alternate flow 3 Alternate flow 1 Alternate flow 2

Scenario 7 Basic flow Alternate flow 4

Scenario 8 Basic flow Alternate flow 3 Alternate flow 4

6. Verification of identification and description of

all potential scenarios for each UC.

In short, each scenario represents a number of

possibilities to execute a UC and prevents from

testing only some potential combinations.

4.2 Phase 2: TCs’ Identification

This phase takes the following activities, which

should be assigned to the test designer:

1. Ideas for new tests are organized based on items

to be tested: functionality (UC), quality attributes,

validation of inflow and outflow, databases,

interfaces, etc. This will depend on the type of

application, technological restrictions, scope of the

project, purpose and motivation of tests, and

expertise of the test team (especially the tester’s).

2. Table 3 must be filled in. This represents the

second device generated from the proposed method

and its data is associated to consider the TC for

Scenario 02-02 (login error). Based on information

generated by the test ideas, there is one TC for

validating a “login error” upon entering invalid

characters; one TC for logins in lowercase letters;

one TC for logins lengths over 10 characters; and

one TC for blank logins. The, information related to

IMPROVING SOFTWARE TEST STRATEGY WITH A METHOD TO SPECIFY TEST CASES (MSTC)

161

all identified TCs is completed: test case ID, TC

name, expected results (values, error messages,

etc.), test level and test type. With respect to the

TC’s ID, we suggest including in the standard

nomenclature determined by the organization a UC-

scenario-TC structure, for instance, in order to

identify that 02-02-01 refers to TC 01 of scenario 02

of UC 02.

3. It is verified that all TCs have been identified for

each scenario. Then, the following phase is

addressed.

4.3 Phase 3: TCs’ Specifications

One of the most significant contributions of this

research is the third device (Design) used to describe

TCs in detail, as shown in Table 4: TCs’

specifications (TCS). This should also be completed

by the test designer and the following activities

should be performed for each TC:

1. Identification of the system’s name, UC ID,

requirement ID, scenario ID, TC ID and TC version.

This allows for laying out a bidirectional trace

between these elements: for instance, it may

determine whether all TCs were specified for the

UCs and whether all UCs were already tested (tests

scope).

2. Identification of the level and type of test

associated to the TC, the information of which is

generated by TC as per the scenarios table.

3. Identification of the test environment. The name

of the company might be indicated, provided that it

is the development or production environment. If

the company has different environments, the

environment where this particular TC will be

executed should be indicated.

4. Identification of the TC creator and tester. We

recommend that two different individuals perform

these activities so the testing process can be true

and reliable.

5. Indication of the TC’s origin date and execution

date.

6. Identification of the conditions that should be

present to execute the TC. Which are the conditions

for causing or making a user to execute a specific

event or series of events? In Table 4, we observe

that all functionalities associated to user’s validation

should be implemented. Likewise, it should be

verified that data to be used for this TC has been

validated and approved by the corresponding level,

etc.

Table 2: Scenarios for UC002.

Scenario ID Originary flow Alternate flow Alternate next Alternate flow

02-01 Basic flow

02-02 Basic flow Alternate flow 1:

02-03 Basic flow Alternate flow 1:

Alternate flow 3:

Cancel Press

02-04 Basic flow Alternate flow 2:

Password Error

02-05 Basic flow Alternate flow 2:

Password Error

Alternate flow 3:

Cancel Press

02-06 Basic flow Alternate flow 1:

Alternate flow 2:

Password Error

02-07 Basic flow Alternate flow 1:

Alternate flow 2:

Password Error

Alternate flow 3:

Cancel Press

02-08 Basic flow Alternate flow 3:

Cancel Press

Scenario ID Originary flow Alternate flow Alternate next Alternate flow

02-01 Basic flow

02-02 Basic flow Alternate flow 1:

02-03 Basic flow Alternate flow 1:

Alternate flow 3:

Cancel Press

02-04 Basic flow Alternate flow 2:

Password Error

02-05 Basic flow Alternate flow 2:

Password Error

Alternate flow 3:

Cancel Press

02-06 Basic flow Alternate flow 1:

Alternate flow 2:

Password Error

02-07 Basic flow Alternate flow 1:

Alternate flow 2:

Password Error

Alternate flow 3:

Cancel Press

02-08 Basic flow Alternate flow 3:

Cancel Press

Table 3: TCs per scenario 02-02.

Test Case

TC name Expected results Test level Test type

02-02-01 Login with invalid

characters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-02 Login with Lowercase

letters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-03 Login length over 10

characters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-04

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

Test Case

TC name Expected results Test level Test type

02-02-01 Login with invalid

characters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-02 Login with Lowercase

letters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-03 Login length over 10

characters

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

02-02-04

Message 20 Error:

Invalid Login

System/

acceptance

Function/ Access

Control

ICEIS 2008 - International Conference on Enterprise Information Systems

162

Table 4: TCs’ Specification (TCS) 02-02-02.

System/Project ID/Name: SIS-PROJ Test Level: System/Acceptance

Use Case ID: CU-02 User’s Validation Test(s) Type(s): Function/ Access Control

Requirement ID:

(solo para Caso de Uso No Funcional)

Test environment:

AMBIENTE1

Scenario ID/Name: 02-02 Login Error Test Case´s Author: LISI

Test Case ID/Name: 02-02-02 Login with Lowercase letters

Tester´sname: Probador 1

Test Case Version: v.1. Origin Date: 10-01-07 Execution Date: 15-03-07

Condition(s) for that Test Case is executed

The user wishes enter the system. All functions related to user´s validation have been implemented. Data to be used for the

tests have been validated and approved. Certain users have been registered as valid users.

For the execution of Test Case:

Step Condition Value(s) Expected Results Obtained Results

It enters the login Log-in attempt

and presses OK.

aDM22

Message 20 Error: Invalid Login

√

It enters the login Log-in attempt

and presses OK.

administrator

Message 20 Error: Invalid Login

√

It enters the login Log-in attempt

and presses OK.

AdminisTRATOR

Message 20 Error: Invalid Login

√

Test Case Approval Criterion : If the expected results are achieved in a 100%

Test Case´s Decision of approval: Approved: X

Failed:

(mark with an X the results obtained)

Test Case´s Date of Approval:_15-03-07_

System/Project ID/Name: SIS-PROJ Test Level: System/Acceptance

Use Case ID: CU-02 User’s Validation Test(s) Type(s): Function/ Access Control

Requirement ID:

(solo para Caso de Uso No Funcional)

Test environment:

AMBIENTE1

Scenario ID/Name: 02-02 Login Error Test Case´s Author: LISI

Test Case ID/Name: 02-02-02 Login with Lowercase letters

Tester´sname: Probador 1

Test Case Version: v.1. Origin Date: 10-01-07 Execution Date: 15-03-07

Condition(s) for that Test Case is executed

The user wishes enter the system. All functions related to user´s validation have been implemented. Data to be used for the

tests have been validated and approved. Certain users have been registered as valid users.

For the execution of Test Case:

Step Condition Value(s) Expected Results Obtained Results

It enters the login Log-in attempt

and presses OK.

aDM22

Message 20 Error: Invalid Login

√

It enters the login Log-in attempt

and presses OK.

administrator

Message 20 Error: Invalid Login

√

It enters the login Log-in attempt

and presses OK.

AdminisTRATOR

Message 20 Error: Invalid Login

√

Test Case Approval Criterion : If the expected results are achieved in a 100%

Test Case´s Decision of approval: Approved: X

Failed:

(mark with an X the results obtained)

Test Case´s Date of Approval:_15-03-07_

7. Description of the TC procedure. This procedure

comprises steps to be taken for testing the UC

scenario through the TC approach, i.e. particular

conditions that might apply for a given step, values

used, results expected and results obtained. It should

be mentioned that the latter is included in the TCS

table upon TC execution.

If data is not properly entered, it would not be

possible to execute tests and determine the results.

Supplementary specifications should be followed to

determine the performance measures (minimum and

maximum), inception valid ranks, interface

protocols, among others.

8. Indication of TC’s approval criterion. As

observed in Table 4, the main criterion is that all

expected results must be 100% achieved.

9. The test analyst and designer verify that all TCs

have been properly specified.

4.4 Phase 4: TCs’ Execution and

Approval

Activities in this phase include the following:

1. Verification that the environment and conditions

to execute the TC are appropriate. Individuals

involved in these tests must cooperate to this

procedure.

2. The test manager and designer must authorize the

test cycle activation.

3. The tester executes all TCs and enters data of the

results obtained into each TCS Table.

4. The test manager decides on the

approval/rejection of a TC based on the criterion

established, and it should also indicate the date of

approval and, in certain cases, its signoff.

5. The test team verifies whether the test cycle

completion criterion was met to decide on the test

cycle’s fully approval or request the application of

additional tests on certain TCs for a subsequent test

cycle, until all acceptance criteria are satisfied.

6. The test team keeps all deliverables and posts the

results of the test cycles, changes, etc., obtained

during the testing process.

4.5 Phase 5: Recording and Analysing

Results

The objective is to maintain and improve the TCs

asset. This is important especially if the intention is

future reuse, in subsequent test cycles or to other

software products. This phase is centered on: (a)

determine the minimal set of additional TCs to

validate the stability of subsequent Builds, (b)

remove TCs that no longer serve a useful purpose or

have become economic infeasible to execute, (c)

conduct general maintenance of and make

improvements to the maintainability of TCs

automation, (d) explore opportunities for reuse and

productivity improvements, (e) maintain test

environment configurations and test data sets, and

(f) document lessons learned –both good and bad

practices–discovered during the TCs execution.

IMPROVING SOFTWARE TEST STRATEGY WITH A METHOD TO SPECIFY TEST CASES (MSTC)

163

4.6 Strategy Checkpoints

To ensure the proper method application and

strategy accomplishment, the following checkpoints

are required:

• Phase 1: (a) Is there any matrix of scenarios per

each system UC?, (b) Check that all scenarios in

the corresponding UC have been included in the

matrix of scenarios per UC, (c) Check for

completeness of IDs of UCs and TCs and their

correspondence with the nomenclature proposed.

• Phase 2: (a) Is there any matrix of TCs per

scenario?, (b) Check for completeness and

accuracy of IDs, names, expected results, tests

levels and test type for each TC matrix per

scenario

• Phase 3: (a) Is there any TC table with

specifications for each TC identified in the prior

stage?, (b) Check traceability among IDs of the

TCs, UCs, their requirements and scenarios, (c)

Check for accuracy and completeness of all

items indicated in the TC specification table, (d)

Were approval criteria for each TC specification

validated?

• Phase 4: (a) Were results from the execution of

TCs through field fill-in, namely obtained

results, approved/failed, date of approval, date

execution, documented?, (b) Was the test cycle

completion criterion indicated in the Test Plan

document?, (c) Were the results from tests and

changes applied during the testing process

delivered and posted?

5 RESULTS DISCUSSION AND

FUTURE WORK

The MSTC method proposed was used for 4

projects, thus obtaining significant results as to: (1)

quality of the developed products. Upon completion

of the construction phase, software systems had

already reached 90% of the expected quality; (2) the

largest project implemented 51 UCs and required

the documentation, execution and approval of 460

TCs. The TCS Table can be used to define TC

procedures associated to non-functional

requirements; and, (3) this experience established a

precedent for future projects, and defined

management indicators that may reflect, for

instance, the average number of TCs per application.

Currently, MSTC is being used by other public and

private sector organizations working in 16 projects;

therefore, the following step in this research should

be posting the results from the method application at

each organization.

6 CONCLUSIONS

This Article described a method for specifying TCs

based on UCs, by incorporating elements to verify

and validate traceability among requirements

management, analysis & design, and tests. In

addition, it evidenced that test costs might be

reduced at a mid- and long-term, since we may

resort to non-specialized testers, provided that what

will be tested, when and how? is clearly defined in

advance.

REFERENCES

Kruchten P., The Rational Unified Process as

Introduction (2nd Edition, Addison – Wesley, 2000.

Leffingwell D. and Widrig D., 2006. Managing Software

Requirements, a Use Case Approach (Second Edition,

Addison-Wesley, Pearson Education, 2006.

Grimán A., Pérez, M. and Mendoza L., 2003. Estrategia

de pruebas para software OO que garantiza

requerimientos no funcionales. III Workshop de

Ingeniería de Software, Chillán, Chile. 1-10.

Lewis W. 2000. Software testing and continuous quality

improvement 000 by CRC Press LLC Auerbach is an

Publisher of CRC Press LLC.

Perry W. 2006. Effective Methods for Software Testing.

Wiley. Third Edition.

Pfleeger S. 1998, Software Engineering, Theory and

Practice, Pretice-Hall.

Pinkster I., Burgt B., Janssen D. and Veenendaal E. 2006.

Successful Test Management. Springer and

Logicacmg.

Pressman R. 2002. Ingeniería del Software: Un enfoque

Práctico, McGraw Hill, 2002).

Sommerville I. 2000. Software Engineering. Pearson

Education.

SWEBOK. 2004. Guide to the Software Engineering

Body of Knowledge 2004 Version. IEEE Computer

Society.

UML Testing Profile Version 1.0 formal/05-07-07. This is

a testing profile for UML 2.0.

Utting M. and Legeard B. 2007. Practical Model-based

Testing. Morgan Kaufmann and Elsevier Publisher.

ICEIS 2008 - International Conference on Enterprise Information Systems

164