Using Technical-Action-Research to Validate a Framework for Authoring

Software Engineering Methods

Miguel Morales-Trujillo

, Hanna Oktaba

and Mario Piattini

KUALI-KAANS Research Group, National Autonomous University of Mexico, Mexico City, Mexico

Alarcos Research Group, University of Castilla – La Mancha, Paseo de la Universidad 4, 13071, Ciudad Real, Spain

Keywords:

TAR, SEMAT, OMG, Case Studies, Evidence-based, Validation, Software Engineering.

Abstract:

The validation of proposals has become a fundamental part of the creation of knowledge in Software Engi-

neering. Initiatives like SEMAT have highlighted the need to base the correctness, usefulness and applicability

of Software Engineering theories and practices on solid evidence. This paper presents the validation process

used for KUALI-BEH, a proposal that became part of an OMG standard. The validation strategy applied was

the result of integrating Technical-Action-Research and Case Study methods. After three years of work, we

can conclude that TAR is a valuable research method, emphasizing that the main advantages of Technical-

Action-Research are continuous feedback and the validation of an artifact, in this case KUALI-BEH, in a real

context.

1 INTRODUCTION

Software Engineering is one of the most knowledge

intensive disciplines (Edwards, 2003; Bjørnson and

Dingsøyr, 2008). However, this particular ﬁeld con-

tains many proposals or theories that have no theoret-

ical rigorousness and have not been adequately vali-

dated in practice.

According to (Johnson et al., 2012) most of these

theories are not subject to serious academic discus-

sion; they are not evaluated or compared as regards

traditional criteria of theoretical quality such as con-

sistency, correctness, comprehensiveness, and preci-

sion.

As (Jacobson et al., 2012) stated “Software Engi-

neering is afﬂicted by a lack of credible experimental

evaluation and validation”. The behavior of the disci-

pline is not that which is desired, thus motivating the

need to transform it and to build theories around it, to

understand it, and more importantly, generate proven

knowledge.

The above fostered the origin of the Software En-

gineering Method and Theory (SEMAT) initiative in

2009, in the form of a call for action to refound Soft-

ware Engineering. Many prominent members of the

Software Engineering community become signatories

and supported the process that would be used to base

Software Engineering on “solid theory, proven prin-

ciples and best practices” (Jacobson et al., 2009).

Since then, the call for action has been trans-

formed into a standardization process that was backed

by the Object Management Group (OMG) through the

Foundation for the Agile Creation and Enactment of

Software Engineering Methods Request for Proposals

(FACESEM RFP) (OMG, 2011), and which two years

later crystallized into a standard: ESSENCE – Ker-

nel and Language for Software Engineering Methods

(OMG, 2013).

In the context of FACESEM, our research group

has actively responded to the RFP by creating a pro-

posal named KUALI-BEH (Morales-Trujillo et al.,

2014b), which later became part of the ESSENCE

speciﬁcation after completing the standardization pro-

cess, which is presented in detail in (Morales-Trujillo

et al., 2014c).

In order to validate KUALI-BEH and evaluate its

usefulness we have, over the last two years, developed

a collaborative workshop and three case studies. The

objective of this paper is to present the process applied

to validate KUALI-BEH.

This paper is organized as follows: Section 2

presents a general background to the research meth-

ods in Software Engineering, focusing on those which

are most relevant for the purpose of this paper. Sec-

tion 3 describes the research strategy applied to create

and validate KUALI-BEH. Sections 4 and 5 show the

Collaborative Workshop and the Family of three Case

Studies. The lessons learned are presented in Section

Morales-Trujillo M., Oktaba H. and Piattini M..

Using Technical-Action-Research to Validate a Framework for Authoring Software Engineering Methods.

DOI: 10.5220/0005338800150027

In Proceedings of the 17th International Conference on Enterprise Information Systems (ICEIS-2015), pages 15-27

ISBN: 978-989-758-097-0

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

6, while the paper concludes in Section 7.

2 RESEARCH METHODS IN

SOFTWARE ENGINEERING

Software Engineering requires both theoretical and

empirical research. The former focuses on foun-

dations and basic theories of software engineering,

whilst the latter concentrates on fundamental princi-

ples, tools/environments, and best practices (Wang,

2007).

As mentioned by (Harrison et al., 1999), Software

Engineering should have strong foundations as a sci-

entiﬁc and engineering discipline. This implies that

validation processes must be sufﬁciently mature to

provide evidence that will support its advances. One

alternative that can be used to achieve this goal is that

of empirical software engineering since, according to

(Belady and Lehman, 1976), this type of research pro-

vides the opportunity to build and verify its theories.

In Software Engineering, the validation must al-

ways involve scaling up to practice, which means that

successive tests take place under increasingly realis-

tic conditions (Wieringa, 2014). No matter what its

form, the essence of an empirical study is the attempt

to learn something useful by comparing theory to re-

ality and to improve our theories as a result (Perry

et al., 2000).

According to (Wang, 2007) and (Genero et al.,

2014), the primary methods used for empirical stud-

ies in Software Engineering encompass: Experi-

ments (Wohlin et al., 2012), Surveys (Kitchenham

and Pﬂeeger, 2008), Case Studies (Yin, 2009; Rune-

son et al., 2012), Action-Research (Medeiros and

Horta-Travassos, 2011), Systematic Literature Re-

views (Kitchenham and Charters, 2007) and Stan-

dardization.

More than one method was considered for the val-

idation of KUALI-BEH, and for the purposes of this

paper, the most relevant research methods are there-

fore described in the following subsections.

2.1 Technical-Action-Research Method

Technical-Action-Research (TAR) is an approach that

is used to validate new artifacts under conditions of

practice (Wieringa and Moralı, 2012). In TAR the re-

searcher uses an artifact in a real world project to help

a client, or gives the artifact to others to use them in a

real world project (Engelsman and Wieringa, 2012).

TAR can be seen as a research method that starts

from the opposite side of traditional research meth-

ods. TAR starts with an artifact, and then tests it un-

der conditions of practice by using it to solve concrete

problems (Wieringa and Moralı, 2012).

In this method the researcher ﬁrst develops the ar-

tifact, then tests it in a hypothetical situation in a lab-

oratory and later scales it to be tested in real world

situations, from an idealized context to the real one.

In technical disciplines, prototypes of artifacts are

ﬁrst tested in the idealized conditions of the labora-

tory, and these conditions are gradually relaxed after

each iteration until a realistic version of the artifact

is tested in a realistic environment, thus allowing the

technical scientist to develop knowledge about the be-

havior of artifacts in practice (Wieringa and Moralı,

2012).

These iterations are called engineering cycles

(Wieringa and Moralı, 2012) or regulative cycles

(Van Strien, 1997). In the engineering cycle an im-

provement problem is investigated, a treatment with

which to solve the problem is then designed and vali-

dated, its improved version is later implemented, and

ﬁnally the experience with the implementation is eval-

uated and the cycle is executed again (Wieringa and

Moralı, 2012).

Each engineering cycle is composed of four steps:

1. Problem Investigation: during this step the

stakeholders, their goals and the improvement

problem are identiﬁed.

2. Treatment Design: during this step the re-

searcher designs the artifact that will be used as

a treatment in the problem context.

3. Design Validation: after the artifact has been de-

signed, the researcher states the expected effects,

expected value, trade-offs and sensitivity.

4. Treatment Implementation and Evaluation:

this step involves the insertion of the artifact into

the real world. The artifact has to be validated and

the researcher has to evaluate the resulting effects.

TAR is based on the assumption that what the

researcher learns in this particular case will provide

lessons learned that will be usable in the next case

(Wieringa and Moralı, 2012). The expected result

from TAR is the validation of the proposed artifact in

the speciﬁed context, thus satisfying the stakeholders’

needs and giving it value.

2.2 Case Study Research Method

A case study is an intensive investigation and analy-

sis of a particular technology, project, organization,

or environment based on information obtained from a

variety of sources such as interviews, surveys, docu-

ments, test or trial results, and archival records (Wang,

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

2007). Wang establishes that case studies link a the-

ory to practice, which allows conclusions to be drawn

about the suitability of a given method for real world

problems at industrial scales (Wang, 2007).

A case study is, according to (Yin, 2009) an em-

pirical inquiry that investigates a contemporary phe-

nomenon in depth and within its real-life context, es-

pecially when the boundaries between phenomenon

and context are not clearly evident.

The essence of a case study, the central tendency

among all types of case study, is that it tries to illu-

minate a decision or set of decisions: why they were

taken, how they were implemented, and with what re-

sult (Schramm, 1971).

Case studies may be used to validate a theory or

method by means of empirical tests. They are also

useful as regards providing a counter instance for a

generally accepted principle. However, the drawback

of case studies as an empirical method in Software

Engineering is the difﬁculties of data collection and

the generalization of ﬁndings via limited cases, par-

ticularly when they are positive but non-exhaustive

(Wang, 2007).

According to (Runeson et al., 2012) the differ-

ences between Software Engineering study objects

rely on the fact that: (i) they are entities that develop

software rather than using it; (ii) they are project-

oriented rather than function oriented organizations;

and (iii) the work studied is advanced rather than rou-

tine engineering. The generic phases of a case study

described by (Runeson et al., 2012) are:

1. Case Study Design: objectives are deﬁned and

the case study is planned.

2. Preparation for Data Collection: procedures

and protocols for data collection are deﬁned.

3. Collecting Evidence: data collection procedures

are executed on the studied case.

4. Analysis of Collected Data: data analysis proce-

dures are applied to the data.

5. Reporting: the study and its conclusions are

packaged in feasible formats for reporting.

3 RESEARCH STRATEGY

APPLIED

The research strategy used to validate KUALI-BEH

was guided by the integration of TAR and Case Study

methods. The TAR method was applied as a frame-

work in which the evaluation during engineering cy-

cles was carried out by applying the Case Study

method.

The research process consisted of ﬁve engineer-

ing cycles (see Figure 1). The validation of these

engineering cycles was carried out by means of case

studies primarily developed in real world software de-

velopment organizations. The artifact was also in-

serted in circles formed of practitioners and experts

from academy and industry, such as a collaborative

academy-industry workshop and the OMG Analysis

and Design Task Force in charge of developing IT

standards.

The ﬁrst engineering cycle (from August 2011

through February 2012) consisted of identifying the

common concepts used in the context of a software

project and was carried out using a literature review.

During the second engineering cycle (from March

through August 2012), an initial version of the arti-

fact was released and validated by means of a col-

laborative workshop, whose participants were active

Software Engineering practitioners, Master’s degree

students and researchers from the discipline.

The third engineering cycle was developed from

September through December 2012. A ﬁrst case

study was carried out during these months in which

an improved version of the artifact was validated in

a Mexican enterprise in charge of software develop-

ment and hardware construction. The artifact was also

evaluated by the OMG task force.

During the fourth cycle (January through July

2013), improvements and lessons learned from the

previous cycle were applied to the artifact. It was also

used in a second case study, which took place in an en-

tity that specializes in requirements speciﬁcation and

software projects design.

During the ﬁfth and the last engineering cycle,

which took place between August 2013 and April

2014, the artifact was enriched and an Authoring

Extension was designed, which now belongs to the

ESSENCE OMG formal speciﬁcation. At that point

the artifact was newly validated by the OMG task

force, while a third case study was simultaneously

carried out in a very small entity in charge of software

development.

A detailed and in-depth description of the valida-

tion of the artifact is presented in Sections 4 and 5.

4 COLLABORATIVE

WORKSHOP

KUALI-BEH was validated by means of a collabora-

tive workshop attended by active practitioners from

industry and academy. This section presents a de-

tailed description of this workshop.

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods

Figure 1: Engineering cycles of the applied research process.

4.1 Problem Investigation

The propositions of the study were focused on the

pertinence, appropriateness and proﬁciency of the

KUALI-BEH common concepts. These propositions

were evaluated by asking the following questions:

• Are the common concepts pertinent?

• Are the deﬁnitions of the common concepts ap-

propriate?

• Are the deﬁnitions of the common concepts simi-

lar to real world usage?

• Are the common concepts proﬁcient?

These questions helped us to support the decisions

made during the acceptance/rejection process carried

out during the identiﬁcation phase.

4.2 Artifact Design

The artifact to be validated by means of the workshop

was the ﬁrst version of KUALI-BEH, which was the

output of the Identiﬁcation phase of this research, and

corresponded to its ﬁrst engineering cycle. This ver-

sion was base lined on February 20th, 2012.

4.3 Design and Implementation

The workshop was divided into eight on-site and on-

line sessions. The theoretical components of KUALI-

BEH were presented as follows:

• Session 1: Motivation, background and overview

of the framework.

• Session 2: The Static view, the deﬁnitions of its

20 common concepts and its relationships and the

graphical representation.

• Session 3: Method properties and the common

concept templates.

• Session 4: The Operational view, the Method En-

actment and the Practice Instance Lifecycle.

• Session 5: The adaptation operations.

• Session 6: The Operational boards.

• Session 7: Putting everything together.

• Session 8: Analysis of results and closure.

During the workshop, three approaches were used

in order to obtain feedback from participants: appli-

cation of surveys, direct interaction with participants

during sessions, which included Q&A runs, and direct

observation of activities. “Homework” activities also

helped analyze the understanding of the proposal.

4.4 Evaluation

The data collected during the workshop resulted in a

set of suggestions that were analyzed and classiﬁed by

the researchers. After this process, the main improve-

ments made to KUALI-BEH concerned the following

issues: (i) The deﬁnitions of terms were enhanced; (ii)

New deﬁnitions and operations were added; and (iii)

Operational rules were improved.

4.5 Results

We developed a collaborative workshop to which or-

ganizations and active software engineers were in-

vited. During the workshop we validated the ﬁrst ver-

sion of KUALI-BEH and we had the chance to try out

its elements one by one.

The feedback received was extremely valuable in-

put with which to improve KUALI-BEH and bridge

the gap between theory and practice. We should

also mention the participants’ active contribution and

involvement which helped establish a conﬁdent en-

vironment in which to share opinions and criticize

KUALI-BEH.

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

This collaborative workshop was the closure to

the second engineering cycle and the suggestions ob-

tained served as a ﬁrst step toward developing an im-

proved version of KUALI-BEH.

5 FAMILY OF CASE STUDIES

In order to prove the usefulness and sufﬁciency of the

framework and its common concepts, KUALI-BEH

was validated using three case studies.

During each case study the practitioners deﬁned

their actual ways of working using KUALI-BEH,

which is referred to as an authoring project. This sec-

tion provides a detailed description of the case studies

and their respective results.

5.1 General Considerations

The research question for the case studies was deﬁned

as follows:

Is KUALI-BEH suitable as regards deﬁning

practitioners’ ways of working during software

projects?

Two more questions also proved to be relevant for

the research:

1. Is the effort of applying KUALI-BEH suitable

when carrying out an authoring project?

2. What is the value obtained by the organization af-

ter having deﬁned its own practices and method?

These questions were used as a basis to deﬁne

the objectives (Os) that are common to the three case

studies, as follows:

• O1: To demonstrate the sufﬁciency of the

KUALI-BEH elements in describing practition-

ers’ ways of working.

• O2: To measure the feasibility of using the con-

cept of Practice to express practitioners’ tacit

practices.

• O3: To identify the value obtained by the or-

ganization as a consequence of deﬁning its own

method composed of its own practices.

In order to decide whether or not the objective had

been achieved, the following indicators (Is) were col-

lected

• I1: Associated with O1, which was collected ap-

plying two surveys to practitioners.

• I2: Associated with O2, which was obtained by

measuring the effort required by practitioners to

document their ways of working.

• I3: Associated with O3, which was obtained by

means of an interview during the Feedback step,

in which practitioners expressed the beneﬁts and

drawbacks of structuring their ways of working

using KUALI-BEH.

Each of the case studies is described in more detail

in the following subsections.

5.2 Case Study 1: InfoBLOCK

The ﬁrst case study (CS1) was carried out at In-

foBLOCK

, a Mexican organization founded in 1997.

InfoBLOCK’s main activities involve hardware con-

struction and software development.

Two of InfoBLOCK’s general managers and two

programmers participated in this case study. The pro-

grammers had spent between 1 and 3 years as active

practitioners and reported having played the roles of

Analyst, Tester, Project Manager and Programmer.

5.2.1 Background

Despite the soundness and constant growth of the or-

ganization, its managers were concerned about con-

trolling the progress of projects in a more detailed

way, but they did not have a deﬁned development pro-

cess. The needs expressed by the organization were

consequently:

• To deﬁne the actual software development process

followed in the organization.

• To be aware of what is being done by the work

team at a particular moment during the develop-

ment process.

The artifact to be validated using CS1 was the sec-

ond version of KUALI-BEH, which was the output of

the second engineering cycle, and had passed through

the Collaborative Workshop presented in Section 4.

5.2.2 Design and Execution

The case study was designed as a sequence of the fol-

lowing seven steps:

1. Presentation of the Case Study

The research team presented the objectives of the

case study to the InfoBLOCK team, after which

KUALI-BEH and its elements were explained.

This step took 90 minutes.

2. Description of Practitioners’ Way of Working

This step was carried out in one work session, dur-

ing which the researcher guided the practitioners

http://www.infoblock.com.mx/

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods

as regards the documentation of their ﬁrst practice

using the KUALI-BEH Practice template.

This also served to gain an insight into the organi-

zation and identify the individual goals and daily

responsibilities of the practitioners involved in the

case study.

After ensuring that the practitioners understood

how to describe their ways of working using the

concept of practice, the researcher assigned them

activities that they should perform independently

during the next step. This step took 120 minutes.

3. Authoring of Practices

The InfoBLOCK team began the authoring of

practices and the running of an internal project si-

multaneously. Throughout this phase, the com-

munication between the team and the researcher

was carried out by videoconferencing and email.

The team completed the following tasks:

(a) The practitioners documented their way of

working using the Practice template. This was

done before they executed the practice in the

internal project.

(b) The researcher then checked aspects of consis-

tency between the KUALI-BEH concepts and

what the team interpreted.

that could be made to the use of concepts but

not the way of working itself.

(d) Finally, the practitioners discussed the sugges-

tions and then generated the 1.0 version of the

practice.

4. Implementation of Documented Practices

Owing to the fact that the case study was de-

veloped alongside a real project, the practitioners

performed the practices almost immediately after

documenting them. This resulted in adjustments

being made to the initial version of the practices,

which were carried out by following the same

tasks developed in the previous step. During the

three weeks of the project, 13 practices were iden-

tiﬁed, documented, implemented and adjusted by

practitioners.

5. Composition of a Method

After the practices had been documented, exe-

cuted and adapted, the practitioners proceeded to

compose a method that represented their actual

way of working using the 13 practices created.

The resulting method was then modiﬁed to

achieve properties of coherency, consistency and

sufﬁciency, thus attaining a well-formed method.

The practitioners used the method template and

the graphical representation of KUALI-BEH to

complete this step.

6. Presentation of Results and Feedback Inter-

view

This step consisted of a session in which the docu-

mented practices and the composed method were

presented to the organization’s members.

During this meeting the researchers collected and

registered orally-expressed lessons learned from

this particular case study, such as suggestions for

improvement, and the beneﬁts and drawbacks of

KUALI-BEH. The InfoBLOCK team also had the

opportunity to think about their actual way of

working and the improvements to be made in fu-

ture projects.

7. Closure

At the phase of closure, the products eventually

generated and the case study report were delivered

to the InfoBLOCK team.

5.2.3 Analysis

At the end of this case study a method composed of 13

practices using KUALI-BEH was documented. Ac-

cording to the initial objectives of this case study the

following results were obtained:

• O1. Based on the data collected by means of sur-

veys and practitioners opinions, it was concluded

that the elements of which KUALI-BEH consists

were sufﬁcient to describe the InfoBLOCK prac-

titioners’ way of working during the real project

selected.

Moreover, it was possible to observe that the com-

mon concepts were everyday concepts for practi-

tioners, who used them in a natural and straight-

forward manner.

• O2. After considering the time and effort required

by practitioners to understand KUALI-BEH, we

were able to conclude that making practitioners’

ways of working explicit in a short period of time

was feasible. Except for the ﬁrst practice, which

required the researcher’s support in documenting

it, 12 more practices were expressed by the prac-

titioners themselves without having to rethink the

common concepts.

The total effort required was 10 hours, resulting

in an average of less than 50 minutes per practice.

It was also observed that a practice contained an

average of between 3 and 4 activities.

Finally, documenting their way of working al-

lowed the practitioners to identify work products

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

that were generated during the project. A total of

28 work products were identiﬁed and grouped into

9 categories.

• O3. The organization obtained the ﬁrst version

of its actual software development method, al-

lowed its managers to share it with others, and

also planned to use it as the means to train new

employees. More importantly, the managers now

have a means to manage progress and control the

projects using the authored practices, thus making

it easier to distribute and measure the work.

5.2.4 Feedback and Improvements

The most important improvement made to the

KUALI-BEH proposal was the adjustment of the

practice template in order to make it more legible, and

these changes were focused on the visual organization

of its elements.

5.3 Case Study 2: Entia

The second case study (CS2) was developed at Entia

a Mexican IT company that has been present in the

industry since 2003 and has 20 employees. With the

motivation to increase its projects’ success rate, En-

tia developed ActiveAction, a game-centered inten-

sive workshop which allows the organization to carry

out the inception phase of its projects.

The general manager and a coach from Entia and

two assistants from the research team took part in this

case study.

5.3.1 Background

The game nature of ActiveAction and the constant on-

the-ﬂy adaptations had led to a rapid evolution of the

workshop, one of whose consequences was the lack

of documentation related to the workshop, while all

the knowledge and techniques belonged to the people

in charge and not to the organization.

The objective deﬁned by the Entia general man-

ager was therefore:

• To document each step of the workshop activities.

The artifact to be validated using CS2 was the

third version of KUALI-BEH, the output of the third

engineering cycle, which involved CS1.

5.3.2 Design and Execution

The generic design of the family of case studies steps

was adapted according to Entia’s context. This case

http://www.entia.com.mx/

study was therefore divided into the following six

steps:

1. On-site Observation of the Workshop

An on-site observation was conducted in order

to understand the new game-based technique, its

purpose and execution.

It is important to mention that an ActiveAction

workshop takes between 10 and 12 hours, signi-

fying that this step took 720 minutes.

2. Presentation of the Case Study

The objectives of the case study and KUALI-BEH

were explained to the participants. This step took

60 minutes.

3. Description of Practitioners’ Way of Working

This step was identical to that in CS1, with the

difference that the researcher, the general manager

and the two assistants participated in it. This step

took 60 minutes.

4. Authoring and Adaptation of Practices

During this phase the assistants identiﬁed the

practices involved in the workshop, completing

the same tasks deﬁned in Step 3 of CS1, with the

only difference that the assistants presented the

practices to the general manager, who suggested

modiﬁcations, which were applied and then gen-

erated using the 1.0 version of the practice. Dur-

ing the three months of the project, 19 practices

were expressed by the assistants and were agreed

upon by the game expert.

The challenge of this step was to identify the in-

puts and results of each practice, since Entia used

to manage all of them as one work product: a

mind map.

5. Composition of a Method

Having documented the practices, the method

composition step was a relatively easy task, be-

cause it resulted in the practices being ordered one

after another, like a waterfall approach.

6. Presentation of Results and Closure

This step consisted of a session in which the docu-

mented practices and the composed method were

presented to the Entia general manager.

The main result of this case study was the “un-

usual” method and its practices, which was pre-

sented as a research paper (Morales-Trujillo et al.,

2014a) during the 9th International Conference on

Evaluation of Novel Approaches to Software En-

gineering 2014 (ENASE14) which took place in

Lisbon, Portugal.

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods

5.3.3 Analysis

At the end of this case study a method composed of 19

practices was documented using KUALI-BEH. Ac-

cording to the initial objectives the following results

were obtained:

• O1. The data collected by means of the sur-

veys and the assistants’ opinions allowed the re-

searchers to conclude that the elements of which

KUALI-BEH is composed were sufﬁcient to de-

scribe the game-based method.

• O2. In this case study the required authoring ef-

fort was 26 hours, resulting in an average of 82

minutes per practice. It was concluded that ex-

pressing the workshop practices in a short period

of time was feasible.

• O3. The only objective deﬁned by Entia’s general

manager was that of documenting each step of the

workshop activities, and this was fully achieved.

Moreover, documenting the method reduced the

possibility of variation and allowed Entia to iden-

tify ways in which to improve it and, more impor-

tantly, replicate it in afﬁliates.

5.3.4 Feedback and Improvements

On the one hand, this case study beneﬁted Entia and

allowed it to achieve important business goals in or-

der to remain competitive. By publishing the paper at

ENASE’14 Entia became more visible.

On the other hand, the KUALI-BEH proposal was

improved and its usefulness was demonstrated in a

context in which software development was not the

main purpose.

5.4 Case Study 3: Tic-Tac

The third case study (CS3) was developed at San Luis

Potosi Superior Tech Institute (ITSSLP) in a software

development entity called Tic-Tac. Tic-Tac was part

of the Tech business incubator program. Two pro-

fessors and two recently graduated systems engineers

participated in this case study.

5.4.1 Background

The software development entity was carrying out

projects without having a deﬁned method or process

that could be followed. The objectives deﬁned by the

professors in charge of coordinating the entity were

therefore:

• To document their software development process.

• To train new work team members using the pro-

cess deﬁned.

The artifact to be validated using CS3 was the

KUALI-BEH Authoring Extension, which was base

lined on November 12th, 2012.

5.4.2 Design and Execution

In order to satisfy the needs deﬁned by the ITSSLP

team, this case study added another objective to the

three previously deﬁned in section 5.1. The new ob-

jective was established as follows:

• O4: To measure the effort required to train a new

work team member using the method deﬁned.

The indicator that would demonstrate whether or

not the objective had been achieved was the follow-

ing:

• I4: Associated with O4, which was obtained by

measuring the effort required by the practitioners

to train a new work team member.

1. Presentation of the Case Study

The objectives of the case study and KUALI-BEH

were explained. This step took 60 minutes.

2. Description of Practitioners’ Way of Working

The researcher and the ITSSLP team documented

its ﬁrst practice, as had occurred in CS1 and CS2.

This step took 60 minutes.

3. Authoring of Practices

The ITSSLP team carried out this step in the same

way as that deﬁned in CS1. This step resulted in

23 documented practices.

4. Composition of a Method

After the practices had been documented, exe-

cuted and adapted, the Tic-Tac method was com-

posed. It will be noted that, unlike the other case

studies, the ITSSLP proposed that their method be

divided into phases.

5. Adaptation of the Method

In order improve and adjust the method, some

adaptation operations were applied. The ITSSLP

team adjusted its method using the concatenation

and combination operations deﬁned in KUALI-

BEH.

6. Presentation of Results and Feedback Inter-

view

During this step, the documented practices and the

composed method were presented to the ITSSLP

authorities.

The researchers used a videoconference to collect

and register orally-expressed lessons learned from

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

this particular case study, such as suggestions for

improvement, along with the beneﬁts and draw-

backs of KUALI-BEH.

7. Closure

The products generated and the report of the case

study were delivered to the ITSSLP team.

5.4.3 Analysis

At the end of this case study a method that repre-

sented the ITSSLP software development entity’s way

of working was composed. The 23 authored practices

and a list of required work products was also gener-

ated. According to the objectives of this case study,

the following results were obtained:

• O1. The sufﬁciency of KUALI-BEH elements

as regards describing the practitioners’ way of

working was conﬁrmed. The results collected

by the surveys demonstrated that it was possible

and appropriate for practitioners to express their

tacit practices using KUALI-BEH, and that its el-

ements are understandable and useful.

• O2. In this case study the required effort was 44

hours, resulting in an average of 115 minutes per

practice. The professors in charge of the entity

expressed satisfaction with the effort made.

• O3. A software development method was docu-

mented and served as basis for the execution of

the new software projects carried out by the en-

tity. At that moment, two new projects were suc-

cessfully developed and two new members were

trained and integrated into the team.

• O4. The ITSSLP followed a seven-step strategy

in order to incorporate two new members, and the

time investment was 7 hours. According to the

professors, the strategy’s success was primarily

related to the participants’ knowledge and skills.

5.4.4 Feedback and Improvements

This case study experience allowed us to improve

KUALI-BEH and its Authoring Extension, and it

more speciﬁcally permitted us to better deﬁne the

states and transitions of the Practice Authoring and

Method Authoring Alphas.

We additionally conﬁrmed that two of the adap-

tation operations are suitable and have real meaning

for practitioners, although it is still necessary to ver-

ify their usefulness in other projects.

The results of the case study, expressed by the

work team and professors during the feedback inter-

view, were:

• “We also think that KUALI-BEH guides the prac-

titioner and causes his/her better participation in

a project”.

• “In addition, it will serve to generate a repository

of available methods and practices for different

projects”.

• “Team members can think and rethink about how

they do things”.

5.5 Results

Conducting the case studies allowed us to learn many

lessons, thus permitting us to improve KUALI-BEH

and its validation process. The main lessons learned

from these three case studies and the collaborative

workshop concern the following issues:

A Tool is Required to Support the Practice-

authoring Process. The use of the Track changes

function of the word processor permitted the come-

and-go of authored practices between practitioners

and researchers to become a valuable learning pro-

cess. However, managing the control version strat-

egy of each practice, making adaptations and shar-

ing practices became more difﬁcult processes. At this

point the need for a tool that would automate this pro-

cess became essential

Support from Managers and Commitment from

Practitioners is Necessary. The validation process

is a process that involves many variables that must be

controlled. On the one hand, it requires the organiza-

tion’s interest in the proposal and an understanding of

how it will deal with its particular needs. On the other

hand, the people in charge of applying the proposal

must see beneﬁts reﬂected in their daily work if bias

and reluctance are to be avoided. A two-level support

from managers and practitioners is therefore manda-

tory, and if either of these levels is not committed, the

case study will not be possible.

Finding Suitable Projects in Order to Validate the

Proposal. It is clear that support and enthusiasm are

not the only ingredients needed to carry out a case

study. The proposal needs to be validated in a real

context and in a suitable project. While the research

team is almost always available and open to the idea

of conducting case studies, the organization needs to

estimate the time, effort and risks of being involved

beforehand, which may lead to time matching prob-

lems between the researcher and the organization’s

project.

Identifying Improvements and Adjustments to Ar-

tifact. The case studies have allowed us to improve

many aspects of KUALI-BEH, thus making it a solid

and accepted proposal that now belongs to an interna-

tional standard.

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods

We were also able to establish that the effort made

(see Table 1) permitted a high ROI for both parties:

practitioners and researchers

Table 1: Effort by step in each case study.

CS1 CS2 CS3

On-site observation - 720 -

Presentation of the CS 90 60 60

Way of working description 120 60 60

Authoring of practices 600

Implementation of practices 390 - 2520

Composition of a method 120

Training new members - - 420

Results and Feedback 60 60 60

Closure 30 30 30

Total effort (in hours) 11.5 27.5 52.5

Practices authored 13 19 23

Authoring effort (in hours) 10 26 44

Average effort per practice 46.2 82.1 114.8

5.6 Threats to Validity and Limitations

In order to avoid threats to validity, various factors

were considered in the collaborative workshop and

the three case studies.

• Construct Validity: Multiple data sources were

used in order to provide evidence and respond to

the research question. The sources used were in-

terviews, direct observations, surveys and work

artifacts, thus covering the three degrees deﬁned

by (Runeson et al., 2012).

Moreover, the validity of the construct (or arti-

fact) was ensured since it was created follow-

ing the mandatory requirements requested by the

OMG in the FACESEM RFP. It was also validated

many times by the OMG Analysis and Design

Task Force.

• Internal Validity: The case studies’ re-

sults demonstrated that the objective for which

KUALI-BEH was created was achieved, thus al-

lowing us to accomplish our goals and the partic-

ipants’ needs.

Different causal relations were examined:

– The surveys’ trustworthiness. The data col-

lected with surveys is closely related to the

practitioners’ experience. In these case studies,

the participants’ experience covered the “ju-

niors” through “seniors” classiﬁcation.

– The number of participants. Although the num-

ber of participants is not large, we can state

that the sample is representative of the proﬁle

of practitioners working in small software or-

ganizations.

– The participants’ age, education and expe-

rience. These factors could have affected

whether they were in favor or against KUALI-

BEH. However, the sample of participants was

diverse.

After having analyzed these factors, all of them

were disallowed, and as a result we have been able

to determine that the implementation of KUALI-

BEH in the case studies allowed us to achieve the

case studies’ objectives.

• External Validity: In spite of the limited num-

ber of organizations that participated in the case

studies, we can state that each organization can

be categorized as a typical software developer en-

tity. The three participating organizations shared

the main characteristics of very small entities in

charge of software development endeavors, which

is the target audience of the KUALI-BEH pro-

posal.

The selection of the participants was not inten-

tional: the organizations that participated in the

case studies expressed their interest in participat-

ing.

It is important to highlight that the methods that

were expressed using KUALI-BEH were the ac-

tual ways of working of active practitioners in-

volved in real projects.

We consequently believe that the sample and the

context of this research were representative. What

is more, thanks to the scopes of OMG and SE-

MAT, the results obtained were contrasted with

researchers and practitioners from other countries

who backed it up.

We therefore conclude that the results obtained

make it possible to generalize about the subject

being researched.

• Reliability: The case studies were carried out by

two researchers and the results were constantly

triangulated to third parties, such as colleagues

and members of the research group. The partic-

ipants were additionally informed of the results

and lessons learned, which were and extensively

disseminated in each of the case study reports de-

livered to them.

A detailed plan that served as guidance for the

case studies is also available, thus making its

replication possible.

In order to improve the validity of the case stud-

ies, it was compared with the checklists provided by

(Runeson et al., 2012).

The following approaches were also taken into ac-

count:

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

• The prolonged involvement, mainly in CS2 and

CS3 (3 and 6 months respectively), which allowed

us to develop a trusting relationship with the par-

ticipants, thus making the collection of data an un-

obstructed process and, in general, to develop the

case studies in a pleasant atmosphere.

• During CS2 we had two assistants who partici-

pated in the data collection process, thus allow-

ing us to analyze different data sources, such as

interviews, surveys and direct observations. This

circumstance made triangulation possible.

• Peer debrieﬁng took place in all three case studies,

owing to the fact that the case studies were carried

out by two researchers. In addition, the ﬁndings

and results were periodically discussed with the

other members of the research group.

• The work products and documents generated dur-

ing each case study were given to the participants

so that they could review them. This took place at

minimum after the ﬁnalization of each step of the

study, but the member checking action was carried

out at any moment if deemed to be necessary.

• A version control strategy was deﬁned from the

very beginning of each case study. Given the tech-

nological background of all the participants, the

audit trail mechanism was easy to follow and its

application was very successful.

Finally, the limitations of the case studies can be

summarized in two points:

• The sample size is small, which therefore limits

the power of generalization. It is necessary to

replicate the case studies with bigger populations.

• Bias in the case studies could have occurred and

been related to the participants’ feeling of being

observed and evaluated. This may have led to an

alteration in their actual way of working.

6 LESSONS LEARNED

After three years of continuous work, we can con-

clude that the TAR is a valuable research method for

the purposes of this research. Its application, in com-

bination with case studies, allowed us to validate and

improve KUALI-BEH after each engineering cycle.

The iterative approach of TAR allowed us to ob-

tain continuous feedback about and validation of the

artifact in a real context and these are, in our opinion,

the main advantages of the TAR.

As researchers we had the opportunity to develop

the three roles identiﬁed by Wieringa: Designer,

Helper and Researcher. Starting as designers, we cre-

ated an artifact whose objective was to resolve a type

of problems present in the industry. During the en-

gineering cycles we inserted the artifact into organi-

zations which were affected by this type of problem.

In order to apply the proposed treatment, and so act-

ing as helpers, we used the artifact and assessed its

functioning.

As researchers we later took advantage of the

lessons learned and analyzed the resulting effects, the

value achieved and the trade-offs obtained with the

objective of adjusting and improving the artifact.

Validating the artifact in a real context allowed us

to gain experience and generate knowledge through

the lessons learned which, according to (Endres and

Rombach, 2003), are the principal means of obtaining

knowledge in the Software Engineering discipline

TAR also provided the organizations involved in

this research with beneﬁts. At the end of the case

studies we were able to establish that:

• Each organization had achieved the stated objec-

tives.

• The practitioners had been trained in a new tech-

nology, in this case KUALI-BEH, at no “extra”

cost.

• The case study results were used to achieve busi-

ness goals, signifying that the artifact was useful

and applicable to their particular contexts.

• Collaboration and partnership ties were promoted

between the university and organizations.

Finally, we can deﬁne TAR as a research method

which gives researchers a valuable opportunity to

learn and obtain knowledge by applying theories in

practice in order to solve real problems by solving real

problems.

7 CONCLUSIONS

The research strategy presented in this paper is the re-

sult of the integration of TAR and Case Study meth-

ods. We took advantage of the engineering cycles,

which were used as the means to create and validate

the artifact in a real context, thus allowing us to mini-

mize the disadvantages of a full empirical study which

can, according to (Harrison et al., 1999), be very dis-

ruptive and time-consuming for the company involved

On the one hand, making the artifact available to

critical reference groups (mainly OMG and SEMAT)

formed by theoreticians and practitioners provided us

with fruitful feedback and a wide range of points of

view. That knowledge became lessons learned, which

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods

were applied during the treatment design phase of

each engineering cycle.

On the other hand, the application of case studies

was meaningful as regards establishing the sensitiv-

ity of the artifact and whether or not it satisﬁed the

stakeholders’ expected value and the hypothesis of the

present research in a real world scenario.

This research strategy allowed us to generate the

KUALI-BEH framework, guided us through the val-

idation process and helped us achieve the objective

and goals set for this research.

Moreover, valuable lessons related to the OMG

standardization process were learned and reported in

(Morales-Trujillo et al., 2014c).

Finally, we can conclude that the combination of

TAR and Case Study research methods was a success-

ful experience, allowing us to validate and improve

KUALI-BEH in several ways and making us realize

that TAR is a powerful means to bridge the gap be-

tween academy and industry.

ACKNOWLEDGEMENTS

This work has been funded by GEODAS-BC

project (Ministerio de Econom

ıa y Competitividad

and FEDER, TIN2012-37493-C03-01); GLOBALIA

project (Consejer

ıa de Educaci

on, Ciencia y Cultura

(Junta de Comunidades de Castilla La Mancha) and

FEDER, PEII11-0291-5274); SDGear project (TSI-

100104-2014-4), framed under the ITEA 2 Call 7,

and co-funded by “Ministerio de Industria, Energ

ıa

y Turismo (Plan Nacional de Investigaci

on Cient

ıﬁca,

Desarrollo e Innovaci

on Tecnol

ogica 2013-2016) and

FEDER”; the Graduate Science and Engineering

Computing (UNAM) and CONACYT (M

exico).

REFERENCES

Belady, L. and Lehman, M. (1976). A model of large pro-

gram development. IBM Syst. J., 15(3):225–252.

Bjørnson, F. O. and Dingsøyr, T. (2008). Knowledge man-

agement in software engineering: A systematic review

of studied concepts, ﬁndings and research methods

used. Inform. Software Tech., 50(11):1055 – 1068.

Edwards, J. S. (2003). Managing software engineers and

their knowledge. In Managing software engineering

knowledge, pages 5–27. Springer.

Endres, A. and Rombach, D. (2003). A Handbook of Soft-

ware and Systems Engineering: Empirical Observa-

tions, Laws and Theories. Fraunhofer IESE series on

software engineering. Pearson/Addison Wesley.

Engelsman, W. and Wieringa, R. (2012). Goal-oriented

requirements engineering and enterprise architecture:

Two case studies and some lessons learned. In Pro-

ceedings of the REFSQ’12, volume 7195 of LNCS,

pages 306–320. Springer-Verlag.

Genero, M., Cruz-Lemus, J., and Piattini, M. (2014).

etodos de Investigaci

on en Ingenier

ıa del Software.

RA-MA Editorial.

Harrison, R., Badoo, N., Barry, E., Bifﬂ, S., Parra, A., Win-

ter, B., and Wst, J. (1999). Directions and method-

ologies for empirical software engineering research.

Empirical Software Engineering, 4(4):405–410.

Jacobson, I., Meyer, B., and Soley, R. (2009). The SEMAT

initiative: A call for action.

Jacobson, I., Ng, P.-W., McMahon, P., Spence, I., and Lid-

man, S. (2012). The essence of software engineering:

The SEMAT kernel. Queue, 10(10):40–51.

Johnson, P., Ekstedt, M., and Jacobson, I. (2012). Where’s

the theory for software engineering? IEEE Softw.,

29(5):96.

Kitchenham, B. and Charters, S. (2007). Guidelines for per-

forming systematic literature reviews in software en-

gineering. Technical report, EBSE-2007-01.

Kitchenham, B. and Pﬂeeger, S. (2008). Personal opinion

surveys. In Shull, F., Singer, J., and Sjberg, D., editors,

Guide to Advanced Empirical Software Engineering,

pages 63–92. Springer London.

Medeiros, P. and Horta-Travassos, G. (2011). Action re-

search can swing the balance in experimental software

engineering. Advances in Computers, pages 205–276.

Morales-Trujillo, M., Oktaba, H., and Gonz

alez, J. (2014a).

Improving software projects inception phase using

games: Activeaction workshop. In Proceedings of the

ENASE’14, pages 180–187.

Morales-Trujillo, M., Oktaba, H., and Piattini, M. (2014b).

Bottom-up authoring of software engineering meth-

ods and practices. IET Software. Submitted.

Morales-Trujillo, M., Oktaba, H., and Piattini, M. (2014c).

The making-of an OMG standard. Computer Stan-

dards & Interfaces. Submitted.

OMG (2011). A foundation for the agile creation and enact-

ment of software engineering methods RFP. Technical

report, Object Management Group, Needham, USA.

OMG (2013). ESSENCE – kernel and language for soft-

ware engineering methods. Technical report, Object

Management Group, Needham, USA.

Perry, D., Porter, A., and Votta, L. (2000). Empirical studies

of software engineering: A roadmap. In Proceedings

of the ICSE’00, pages 345–355. ACM.

Runeson, P., Host, M., Rainer, A., and Regnell, B. (2012).

Case Study Research in Software Engineering: Guide-

lines and Examples. Wiley.

Schramm, W. (1971). Notes on Case Studies of Instruc-

tional Media Projects [microform] / Wilbur Schramm.

ERIC Clearinghouse.

Van Strien, P. (1997). Towards a methodology of psycho-

logical practice the regulative cycle. Theory & Psy-

chology, 7(5):683–700.

Wang, Y. (2007). Software Engineering Foundations: A

Software Science Perspective. Auerbach Publications,

Boston, MA, USA, 1st edition.

ICEIS2015-17thInternationalConferenceonEnterpriseInformationSystems

Wieringa, R. (2014). Empirical research methods for tech-

nology validation: Scaling up to practice. J. Syst.

Softw., 95:19–31.

Wieringa, R. and Moralı, A. (2012). Technical action re-

search as a validation method in information systems

design science. In Proceedings of the DESRIST’12,

pages 220–238. Springer-Verlag.

Wohlin, C., Runeson, P., H

ost, M., Ohlsson, M. C., Regnell,

B., and Wessl

en, A. (2012). Experimentation in Soft-

ware Engineering: An Introduction. Springer-Verlag.

Yin, R. (2009). Case Study Research: Design and Meth-

ods. Applied Social Research Methods. SAGE Publi-

cations, 4th edition.

UsingTechnical-Action-ResearchtoValidateaFrameworkforAuthoringSoftwareEngineeringMethods