Scaling a Standardized Procedure to Conceptualizing and

Completing User Stories across Scrum Teams and Industries

Matthew Ormsby and Curtis Busby-Earle

Department of Computing, University of the West Indies, Mona Campus, Kingston, Jamaica

Keywords: Agile, Scaling Agile, User Story, Documentation, Definition of Done, Steps of Doneness, Scrum, Velocity,

Story Points, Estimating.

Abstract: Agile Software Development (ASD) is a user-centric approach to executing software development projects

in continuous iterations. This approach focuses on collaboration and communication among self-organizing,

cross-functional teams. Over the last two decades, ASD has been viewed as the answer to the pitfalls of the

Waterfall Software Development approach. As such, large organizations have embraced this methodology

and its scaling frameworks to produce faster to-market software. It is important to note that the agile

methodology does not guarantee that all organizational and project problems related to software development

are solved. This paper aims to contribute to the agile community by testing an existing procedure that has

proven successful within a scrum team of a single company and applying it across multiple scrum teams

within different domains in a large organization.

1 INTRODUCTION

1.1 Background

Kingston, Jamaica is home to one of the largest

organizational groups in the Caribbean. With nearly a

dozen subsidiaries falling under their ambit, their

portfolio includes pension, health and assets

insurance, wealth and asset management/brokering

and banking industries.

The organization has maintained productivity of

its core business while exploring inorganic

opportunities for growth, as an essential component

of its growth strategy. Over the last fifteen years, the

organization has grown and expanded their offerings

(products and services) to deliver a first-class

experience to over a million customers. This focus

strategy has awarded them increased profits year after

year, peaking in the last 3 years with record profits of

over $10 billion JMD at the end of each successive

financial year. Having such a diverse portfolio, the

organization’s compliance framework and

governance structure over the varying industries hold

significant weight in project execution of new and

enhanced offerings: banking regulations, Payment

Card Industry Data Security Standard (PCI-DSS) and

securing of personally identifiable information (PII)

data are just a few of the various compliance

regulations and security measures observed.

In pursuit of delivering top-quality offerings to

their customers, the project management arm of the

organization has traditionally applied the waterfall

methodology in project execution. However, the

organization has experienced issues delivering

projects using this methodology. These issues have

resulted in projects being delivered years later than

intended at which point, most of the business value

has already been eroded. Taking such experiences

into consideration, the organization began exploring

ASD.

Within the last decade, there has been a

technology boom that has given rise to an increase in

the rate at which products are launched to market. A

result of the more-iterative approach that ASD offers

along with faster time to market, higher user-

acquisition and user-retention has emerged.

Kisielnicki and Misiak concluded that businesses

today cannot wait an extended period for projects to

conclude (2017). They further concluded that ASD

can give fast return on investment, frequent

functionalities delivery, easy adaptability to changes

when required and cost-effective maintenance.

Ormsby, M. and Busby-Earle, C.

Scaling a Standardized Procedure to Conceptualizing and Completing User Stories across Scrum Teams and Industries.

DOI: 10.5220/0007731501270133

In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2019), pages 127-133

ISBN: 978-989-758-375-9

127

1.2 Paradigm Shift

The organization assessed incorporating ASD with

their development methodology. Initial reviews of

ASD showed promise to their key performance

indicators. Consequently, the decision was made to

transition the organization from utilization of the

waterfall methodology to agile. The objective at the

end of the transition was to revolutionize the culture

within the organization in hopes of achieving faster

time-to-market offerings, increased engagement of

staff, higher user retention and overall, increased

profit. Mahadevan, Kettinger and Meservy’s study

highlights the many challenges that exist when

integrating Agile in organizations (2015) and Brizard

posits that transparency is critical to the successful

implementation of ASD (2015). Transparency was

not found to be a traditional strength of the

organization. As such, the transition to ASD would

not be straightforward as it would require a drastic

change in the organization's culture. They

subsequently mobilized Agile consultants to assist

with this transition in the hopes of achieving optimal

results. Besides the banking sector, other sectors have

similar compliance frameworks and regulations to

adhere to that can make innovation a challenge.

Bhargava highlights several regulations that

pharmaceutical software development needs to

adhere to by demonstrating how Agile permitted their

5-sprint project to be delivered on time, within budget

and proper documentation and logs of changes that

were needed for FDA submission (2017). They

conclude that the Agile methodology has positively

impacted the pharmaceutical sector. Agile can be

positively applied to other sectors that are heavily

regulated, such as the banking sector. Kraft concludes

that ASD can benefit federals projects to “deliver

increased value, reduced risk, enhanced visibility and

greater adaptability” (2018).

One of the initial key decisions in the transitional

stage was to identify which Agile methodology would

be the ideal fit for the organization. After careful

consideration, training and consultation, it was

determined that the Scrum framework would provide

the best organizational fit. Consultants, contractors

and newly hired staff were brought in to assist with

the integration of the framework within the

organization. These steps taken by the organization

are important as Kraft highlights that agencies who

are “looking to implement or expand the use of agile

should evaluate their specific circumstances to

determine what approach will work best”. One of the

most important concepts of Agile is the use of co-

located teams. Structures were therefore built and

furnished to accommodate these teams. A large

subset of the current workforce was identified and

trained in the Agile methodology and the Scrum

framework. While this undertaking was aimed at

ensuring the organization effectively integrated Agile

into its core operations, it was also necessary to

ensure that the scaling of this methodology would not

serve as a negative disruptor to its overall operations.

1.3 Scaling Agile

The term, “Agile at Scale” was coined by Agile

industry professionals. It represents an efficient

framework that can be followed when transitioning to

Agile. Spotify and Netflix are two organizations

which have been highly successful at transitioning to

become Agile organizations via the utilization of

Agile at Scale. They have been case studied and the

steps taken at Spotify to transition to Agile have

subsequently been mirrored by thousands of small,

medium and large organizations throughout the

world.

Despite the strong global support for Scaled Agile

Framework® (SAFe)® as highlighted in VersionOne’s

(2018) 12th Annual State of Agile Report, the

organization, in collaboration with consultants,

sought to implement their own Agile at Scale

framework. They contemporaneously took heed of

the failures and successes of other organizations

which had attempted likewise. The report conveyed

the following as the top five (5) challenges Agile

organizations have experienced in both adopting and

scaling Agile:

1. Organizational culture at odds with Agile values

2. General organization resistance to change

3. Inadequate management support and

sponsorship

4. Insufficient training and education

5. Inconsistent processes and practices across

teams

These challenges are highlighted in a study

conducted by Ashmore et al. that demonstrates “there

are important adaptations and cultural differences that

should be considered when an organization starts

leveraging agile methods” (2018).

Armed with this knowledge, the organization

embarked on a transformative journey to become an

Agile organization. Notably, one of the challenges the

organization experienced early in the transformation

process was that the co-located teams were isolating

themselves from the rest of the organization. This

gave rise to an issue of process standardization across

each team. On closer inspection of one of the

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

128

processes, the definitions of done in user story

completion in ASD, it was determined that the issue

did not lie with the definitions but in the procedure

used by each team to achieve it.

1.4 Problem Statement

In response to the procedural problem identified, our

study proposes an investigation to determine the

lacunae in the process and apply Ormsby and Busby-

Earle’s (2018) procedure to four (4) of the scrum

teams within the organization.

The objectives of this study were:

1. To standardize the procedure that was

executed by each team;

2. Verify if Ormsby and Busby-Earle’s

procedure could be successful in scrum teams

of different sizes tackling different domain

problems;

3. Increase each team’s percentage of story

points and the net promoter score for the

solutions that they were each building.

A limitation of Ormsby and Busby-Earle’s study is

that it was executed on one scrum team. The next step

that arose from their study was to have the procedure

incorporated in other technology companies across

teams with varying compositions, development tasks

and levels of skill.

2 EXPERIMENTAL APPROACH

AND COMPUTATIONAL

DETAILS

Ormsby and Busby-Earle referred to lacunae within

the steps of conceptualizing and completing a user

story, and put forth a generic procedure to address

these gaps. This procedure resulted in an average

increase in velocity of 2.81%, as well as, morale

boosts from all stakeholders as their procedure

produced more user-centric user stories and

continuous feedback between the scrum team and

clients. This helped to address Rothman’s concern

that teams do not fully understand what done means

at different levels in the life cycle of a user story as

well as increased the transparency needed for success

(2017).

2.1 Experimental Approach

The following data points on each team were tracked

and recorded: number of story points committed to at

the beginning of a sprint; number of story points

completed at the end of a sprint; and survey

completed by customers in which they rated (on a

scale of 0 – 10) how likely they would recommend

the product to friends or family.

Table 1: Team composition and assigned projects.

Team 1

Team 2

Team 3

Team 4

Description of

solution being

built by team

A product to alleviate the

account opening process.

By alleviating this

process, they hoped to

increase the number of

accounts that were being

opened thus increasing the

available number of

customers that the entire

organization can engage

in cross selling activities.

A solution to digitally

transform the auto

insurance experience for

current and prospective

customers. This

transformation would take

the form of both

redefining the process to

be more efficient and

utilize technology to

produce a convenient and

satisfying experience to

customers.

A platform to deliver a

revamp of the organization

digital solutions. This revamp

will take the many digital

application currently offered

to customers and bring them

under one platform. This

platform will be data driven

to give the organization a

holistic view of its customers

and the products that will

better fit their needs.

A product to drive

the investment

market in a new

direction. This

drive will seek to

revolutionize

investment in the

region by

empowering

existing and

prospective

customers.

Team size

Core

competencies

Web & Mobile

Development, Banking,

Document Management,

Testing, User Experience

and Marketing.

Mobile Development,

Auto Insurance, Payment,

Testing, User Experience,

Marketing and Legal &

Compliance.

Web & Mobile Development,

Auto and Life Insurance,

Banking, Data Analytics,

testing, User Experience,

Marketing and Legal &

Compliance.

Web

Development,

Investment, Data

Analytics, Testing,

User Experience

and Marketing.

Age range of

team members

19 - 51 years old

22 - 37 years old

21 - 45 years old

18 - 36 years old

Nationality of

team members

American, Antiguan,

Indian and Jamaican.

Barbadian, British and

Jamaican

American, Antiguan,

Barbadian, Canadian, Indian,

Jamaican and Trinidadian.

American, Indian,

and Jamaican.

Scaling a Standardized Procedure to Conceptualizing and Completing User Stories across Scrum Teams and Industries

129

From these data points, we derived two main

statistics:

1. Percentage of story points completed –

number of story points completed at the end of

a sprint divided by number of story points

committed to at the beginning of that

corresponding sprint multiplied by 100

2. Net promoter score (NPS) – subtract the

percentage of detractors (customers who gave

a rating of 0 through 6) from the percentage of

promoters (customers who gave a rating of 9

or 10)

Teams used one-week sprints. Each team was also

assigned a designated product owner and scrum

master and all teams were collocated

2.2 Prior Execution of User Stories

Upon the genesis of an Agile team, one of the

necessary action items was that the team was required

to create a definition of done for a user story. The

definition of done detailed the criteria that would

indicate that a user story was completed (a checklist)

and was expected to aid each team in understanding,

accepting and agreeing the point at which a user story

was complete. A team then began sprinting with their

newly-formed definition of done.

Table 2 illustrates the percentage of story points

completed and the NPS score of the solution

following each sprint for the four Agile teams in their

last five sprints prior to the implementation of our

procedure. Two points can be observed from Table 2:

1. The teams were not stable – the percentage of

story points completed and the NPSs never

consistently increased;

2. Only one (1) team, in one (1) sprint was able

to exceed the 80% mark.

Table 2: Percentage of story points completed and the NPSs

for the last five (5) sprints before introduction of the

procedure.

# of Sprint

Before

Team 1

Team 2

Team 3

Team 4

NPS

27.3

72.1

72.7

68.5

46.7

69.2

45.0

82.1

58.8

63.6

63.3

76.2

67.3

76.7

65.5

60.8

76.4

78.3

76.7

56.6

2.3 Introduction of Ormsby and

Busby-Earle’s (2018) Procedure

In the first sprint, following the introduction of the

procedure, there was a drop in the percentage of story

points completed for all four teams. The teams

committed to approximately the same number of

points as they did in their previous sprint. In the

sprints that followed, the percentage of story points

completed steadily and consistently increased. There

was a positive impact with the introduction of the

procedure as the percentages for the teams were able

to surpass the 80% mark. This solved one of the

problems that was previously observed when the

teams used other procedures. However, this did not

solve for the issue of team stabilization. As seen in

Table 3, the percentage of story points completed was

inconsistent. Our procedure did however aid in

customer satisfaction as NPSs improved for all four

solutions that were built by the teams.

Table 3: Percentage of story points completed and the NPSs

after implementation of our procedure.

Sprint

Team 1

Team 2

Team 3

Team 4

NPS

53.5

58.0

43.5

51.7

70.7

68.5

72.7

70.1

80.3

84.7

79.1

82.2

81.3

87.4

80.6

86.7

84.1

85.4

85.7

83.3

86.0

88.2

85.9

84.6

2.4 Adjustment to Ormsby and

Busby-Earle’s (2018) Procedure

To stabilize the percentage of story points completed

and the NPS of the teams, adjustments were made to

the procedure.

In the first sprint following the adjustment (sprint

1), there was an anticipated drop in the performance

of the teams. In sprint 2, there was a notable increase

in percentages of story points completed and the

NPSs. This continued and resulted in the desired team

stabilization in the subsequent sprints.

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

130

Table 4: Percentage of story points completed and the NPSs

after adjustment to the procedure.

Sprint

Team 1

Team 2

Team 3

Team 4

NPS

68.1

73.2

71.1

75.6

88.3

87.4

87.1

87.7

90.1

98.4

93.8

89.3

90.1

98.9

94.6

88.9

91.6

97.1

93.1

87.9

91.2

98.2

95.1

89.2

90.9

100.0

95.2

87.1

92.7

98.1

98.5

90.7

3 ANALYSIS OF ADJUSTMENTS

It can be observed in Table 2 that the percentage of

story points completed of the four teams was very

inconsistent. This was due to the teams learning from

their mistakes and applying corrective measures. The

teams, however, isolated themselves from the rest of

the organization and as such, the lessons learned by

each team were not shared with the other teams. This

engendered within teams a practice of developing

solutions for problems that were already solved by

another team.

It is to be noted that the introduction of our

procedure came at a cost, as depicted in Table 3.

There was an initial drop in the percentage of story

points completed. A steady increase was however

observed in the subsequent sprints. Further, all the

teams were able to surpass the 80% mark and

remained above that mark until the completion of

their respective development projects.

There was also an increase in the NPS after the

introduction of our procedure. Members of the teams

indicated that the new procedure helped them to

coordinate the opportune times to obtain feedback,

i.e. receiving the feedback without distracting the

teams.

There was insufficient stability in the percentage

of story points completed. The organization sought

more stability to be able to more efficiently forecast

the amount of work the teams could manage. Taking

this issue into consideration in addition to the

feedback from the respective teams, we further

investigated our procedure and made the requisite

adjustments. This culminated in the creation of a new

procedure which we have named the ‘Scaled Steps of

Doneness’ (SSOD).

As seen in Figure 1, we took the technical design

step which was included in step 10 of Ormsby and

Busby-Earle’s procedure, within the Technology

grouping, and moved it to an earlier stage in the

procedure within the User Experience group, Step 9.

Moreover, an additional step was added to the

procedure: Step 8 – User Feedback Session.

Embracing these steps in the User Experience group

allowed for earlier feedback from both end-users and

the rest of the Scrum teams. This adjustment

encouraged more collaboration and a superior

understanding of the goal of each user story. There

were also some minor changes in the latter stages of

the procedure. The Agile teams in the organization

had decided that there was no need to gain business

analyst approval. It was determined that with so many

“touchpoints” earlier in the procedure, it was an

unnecessary step and, in some cases, resulted in user

stories not being completed.

Each of the Agile teams implemented the SSOD

and predictably, there was a drop in the percentages

of story points completed during the initial sprint, as

seen in Table 4. This was due to team members

needing time to become familiar with the new method

of working imposed by the SSOD. However, the drop

in the story points completed was not considered

drastic as the Agile teams were already familiar with

most of the steps of the new procedure. In the

proceeding sprints, the percentage of story points

continued to increase and quickly stabilized. In the

case of teams 1, 2 and 3, they exceeded the 90% mark.

Team 4 stabilized in the high 80% but it can be

observed that they broke the 90% mark in sprint 14.

Additionally, the NPSs increased and showed signs

of high success for the teams. This can be observed in

Table 4 where all teams increased their net promoter

scores, three of the teams surpassed the 40 mark and

one of the teams scored a high 67 in the last two

sprints of this study. These scores reflect the

satisfaction by end-users.

4 CONCLUSION

In summary, we have provided empirical evidence

that our SSOD procedure helps to solve one of the key

problems of working in an Agile environment as

stated by the 12th Annual State of Agile report:

inconsistent agile practices and processes. This is

seen in Table 4, whereby having had the teams

understand and implement the procedure, they were

Scaling a Standardized Procedure to Conceptualizing and Completing User Stories across Scrum Teams and Industries

131

Figure 1: Scaled Steps of Doneness (SSOD) Procedure – changes to the procedure are highlighted in bold.

able to experience increased percentage of story

points completed and net promoter scores. Moreover,

it was observed that team morale improved as a result.

Further our SSOD procedure can adapt to teams of

varying skillsets, age and nationality. It should be

noted that the majority of the nationalities was

represented by North America and The Caribbean.

The procedure requires communication among all

team members. A next step would be to apply the

procedure to teams composed of persons speaking

multiple languages.

Ormsby and Busby-Earle’s procedure did not

succour in stabilizing the velocities of the teams. This

lacuna was due to lack of a clear understanding of the

end-users wants as well as the way technology can

facilitate the realization of those wants. The designing

steps in the User Experience group needed to

encompass more feedback steps to truly understand

the feasible wants of the end-users. This would have

resulted in a clear acceptance of the goal of the user

story.

The results observed in Table 4, indicate the

successful impact of the adjustments to the procedure.

As the teams continued to settle into the new process,

they were able to stabilize their percentage of story

points completed and net promoter scores. A product

owner was able to produce more accurate forecasts of

the amount of work a team could complete when

armed with a stabilized percentage of story points

completed. This is important as this enables the

organization to deduce whether their timelines are on

track at an early stage of a project. Earlier feedback

leads to more success.

An observation from the experiment was that the

cost of change is expensive. This was illustrated in

both Tables 3 and 4. At introduction of a new

procedure, there was a corresponding descent in the

percentage of story points completed. Changes are

expensive and should be both deliberate and strategic.

Moreover, communication of a guiding procedure

initially would have also contributed to higher

collaboration among team members and overall,

stronger team health in the long run.

The research of Silva et al. concluded that there is

a need for more and better empirical studies

documenting and evaluating the use of the definition

of done (2017). These results greatly contribute to

further assisting in documenting the definition of

done. The goal of the Agile methodology is to create

and deliver business value expeditiously. Therefore,

if a user story that is done has resulted in bugs within

the production environment, that user story can be

considered incomplete.

The organization has adopted the SSOD

procedure within their culture and have had all

subsequent teams incorporate this procedure before

they begin sprinting. It should also be noted that the

organization has eleven (11) Agile teams that have

integrated our SSOD procedure into their ASD.

ACKNOWLEDGEMENTS

We are immensely grateful to the Caribbean-based

regional company for being adventurous in

incorporating our adjusted procedure into their

ENASE 2019 - 14th International Conference on Evaluation of Novel Approaches to Software Engineering

132

organization. The results have not only benefitted

their organization but yields an outstanding

contribution to the community. Regrettably, due to

Non-Disclosure Agreements, the organization cannot

be named.

REFERENCES

Brizard, T. (2015). Broken Agile: Stories from the

Trenches. 2nd ed. New York: Apress Media LLC.

VersionOne. (2018). 12th Annual State of Agile Report.

Available at: https://explore.versionone.com/state-of-

agile/versionone-12th-annual-state-of-agile-report

[Accessed 17 Oct. 2018].

Rothman, J. (2017). Create Your Successful Agile Project:

Collaborate, Measure, Estimate, Deliver. North

Carolina: The Pragmatic Bookshelf.

Ormsby, M. and Busby-Earle, C. (2017). A Standardized

Procedure To Conceptualizing And Completing User

Stories. In: 2017 International Conference on

Computational Science and Computational Intelligence

(CSCI). Las Vegas: IEEE.

Silva, A., Araújo, T., Nunes, J., Perkusich, M., Dilorenzo,

E., Almeida, H. and Perkusich, A. (2017). A systematic

review on the use of Definition of Done on agile

software development projects. In: EASE'17

Proceedings of the 21st International Conference on

Evaluation and Assessment in Software Engineering.

[online] New York: ACM, pp. 364-373. Available at

https://dl.acm.org/citation.cfm?id=3084226.3084262

[Accessed 30 Aug 2018].

Mahadevan, L., Kettinger, W. and Meservy, T. (2015).

Running on Hybrid: Control Changes when Introducing

an Agile Methodology in a Traditional “Waterfall”

System Development Environment. Communications

of the Association for Information Systems, [online] Vol

36, Article 5. Available at: http://aisel.aisnet.

org/cais/vol36/iss1/5 [Accessed 30 Aug. 2018].

Bhargava, P. (2017). The Agile Approach in

Pharmaceutical Software Development. Honors

Scholar Theses. Available at http://digitalcommons

.uconn.edu/srhonors_theses/523 [Accessed 15 Sep.

2018].

Kraft, C. (2018). Agile Project Management on

Government Finance Projects. The Journal of

Government Financial Management, [Online]. 67.

Available at: https://www.questia.com/library/journal/

1P4-2101226812/agile-project-management-on-gover

nment-finance-projects [Accessed 21 September 2018].

Ashmore, S., Townsend, A., DeMarie, S. and Mennecke, B.

(2018). An Exploratory Examination of Modes of

Interaction and Work In Waterfall and Agile Teams.

International Journal of Agile Systems and

Management (IJASM), 11, 67.

Kisielnicki, J., Misiak, A. (2017). Effectiveness Of Agile

Compared To Waterfall Implementation Methods In IT

Projects: Analysis Based On Business Intelligence

Projects. Foundations of Management, 9.

Scaling a Standardized Procedure to Conceptualizing and Completing User Stories across Scrum Teams and Industries

133