Customer Involvement in the Scaled Agile Framework

Results from a Case Study in an Industrial Company

Jos Trienekens

, Rob Kusters

, Hatta B. Himawan

and Jan van Moll

University of Technology Eindhoven, Den Dolech 2 Eindhoven, The Netherlands

Philips Medical Systems, Veenpluis 4-6, Best, The Netherlands

Keywords: SAFe, Conceptual Model, Customer Involvement, Case Study.

Abstract: The Scaled Agile Framework (SAFe) has emerged over the last years as an approach which supports the

improvement of software and systems development. SAFe claimes solutions for business challenges, such as

shortening cycle’s times, improving product quality, increasing team members’ satisfaction, and involving

the customer in product development. However, regarding customer involvement, there is limited research,

both in SAFe and in real-life agile software development projects. In previous work we developed an initial

conceptual customer involvement model for the SAFe domain in Philips Medical Systems. In this paper this

initial model will be extended and enriched on the basis of a case study in an industrial company.

1 INTRODUCTION

As customer involvement is an essential factor for

developing successful software products (Sauvola et

al, 2015), companies are often not supported in

identifing and selecting the right customer types and

the customer skills that are needed. Consequently,

customers cannot be assigned appropriately in

development processes, and their performance cannot

be measured (

Ghobadi and Mathiassen, 2013). For

instance, a customer can have essential knowledge of

a product, but can lack authority in development

processes to decide for particular product features

(Olson and Bakke, 2001). This can cause declining

customer motivation and loss of customer interest to

get or stay involved in software development. In our

previous work we showed that limited research has

been done in the SAFe domain on how to involve

customers in real-life agile projects (Trienekens et al,

2017). SAFe considers user feedback and the usage

of intrinsic customer knowledge as key for a

successful application (Laanti, 2014). Customers are

considered as having a critical role in the various

aspects of SAFe implementations (SAFe, 2016).

However, although SAFe addresses customer

involvement issues in its framework, there is limited

research done on how to determine and evaluate

customer involvement. In Philips Medical Systems,

the case study environment of our research, medical

embedded software development is carried out in

large evolutionay software development projects

(Turetken et al, 2016). Currently SAFe is being

implemented in this company in various projects in

different departments and business units. Customer

involvement is considered in this company as a

challenging and promising area in SAFe

implementations.

In Section 2, some related work will be discussed, and

we will refer to our initial conceptual customer

involvement model. Section 3 will discuss the

methodology that we followed to extend and enrich

the initial model in a case study in the company. In

Section 4 the case study results will be presented.

Section 5 will cover validation issues, and Section 6

will finalise the paper with conclusions.

2 RELATED WORK

The SAFe framework covers both organizational

levels and processes for agile development practices,

see the “4-level view” in

http://www.scaledagileframework.com/. Four

organizational levels can be recognized, respectively

the Team Level, the Program Level, the Value

Stream, and the Portfolio Level. Although SAFe

states that customers should be empowered in

processes such as requirements management,

defining solutions, planning, demonstration, and

104

Trienekens, J., Kusters, R., Himawan, H. and van Moll, J.

Customer Involvement in the Scaled Agile Framework.

DOI: 10.5220/0006773701040110

In Proceedings of the 20th International Conference on Enterprise Information Systems (ICEIS 2018), pages 104-110

ISBN: 978-989-758-298-1

product evaluations (Leffingwell, 2010), it does not

provide explicit guidance for defining and

implementing customer involvement, for example

with respect to the type of customer to be involved, in

what specific activities, and the customer’s barriers to

overcome (Laage-Hellman et al, 2014). Three

research domains provided a basis for our initial

conceptual customer involvement model, see Figure

1. This figure shows the Scaled Agile Framework

domain as the main research area, and the highly

relevant intersections between the three domains.

Figure 1: Research domains.

Table I is a result of investigating customer

involvement in the SAFe domain. It shows customer

involvement activities on the different levels of the

SAFe framework The Program level and the Value

Stream are merged in this table because customers

have similar activities on these levels, and/or

activities are closely linked. Customer activities on

these levels are related to contributing in planning and

verification and validation and feedback. On the

Team level customers should contribute to

operational development activities such user story

development and functional testing. On the Portfolio

Level no particular customer involvement has been

defined. In agile software development, the structured

evolution of agile methods has been investigated

(Abrahamsson, 2003). Customers should have an

important role in software development processes,

e.g., as product owner with critical tasks, such as

defining product features, reviewing features, and

providing feedback (Schwaber and Beedle, 2002). In

our previous work we identified customer

involvement concepts such as: customer involvement

level, customer specification, customer selection and

customer value optimization (Trienekens et al, 2017).

With developed an initial customer involvement

model to provide a basis for a case study. The goal of

the company was to establish customer involvement

in order to improve product quality and to reduce

uncertainty in projects. A first stage in our initial

model addresses project risk identification. In stage 2,

the results of project risk identification are used to

determine the customer involvement level in the

Table 1: Customer involvement activities in SAFe.

SAFe Level Customer Involvement

Activities

Portfolio -

Program &

Value Stream

Evaluating the full system

produced, including feedback

Contributing in estimating

scope, time, and other constraints

Attending program increment

planning to create plans

Contributing in defining a

roadmap, milestones, and releases

Participating in inspection and

adaption to improve product

performance

Team Contributing in creating user

stories. Performing functional &

system acceptance testing.

project. To support this stage, customer involvement

concepts, such as customer roles (i.e., resource, co-

creator, user) (Nambisan, 2002), and customer

knowledge issues can be applied. Subsequently, the

next three stages of our initial model followed an

approach for involving external parties, as developed

in (Van Weele, 2009). These are respectively, a

specification, a selection, and a customer value

optimization stage. The initial model appeared to be

a quite generic model for the company, on a quite

high level of abstraction. Contradictive opinions

appeared to exist regarding the identified customer

involvement concepts, as mentioned in the foregoing.

For instance for program managers customers should

be involved in SAFe projects independent from risks

involved, or risk determination should be supported

with clear guidelines for characterising projects and

their risks. Product managers considered concepts

such as customer involvement level and customer

specification as not being elaborated sufficiently and

hesistated to involve, on that basis, customers in their

projects. To extend and enrich the concepts in our

initial model, and to increase the acceptability and

applicability of the model it was decided to do a case

study project, in close collaboration with well-

selected experts from different business units in the

company.

3 METHODOLOGY

The methodology of the research project consisted of

three steps.

Step 1: Data collection and analysis.

Customer Involvement in the Scaled Agile Framework

105

In this step the customer involvement concepts as

identified in our previous research have been taken as

a basis for our case study. The objective was to extend

and enrich these concepts in order to improve their

understandability. For the case study representative

experts of different business units have been selected.

We made a distinction between a first data collection

preparation phase and a second ‘Delphi-based’ data

collection phase (Mingers, 2001). Preparation of the

data collection was aimed at ensuring the right

interpretation and relevance of the concepts in the

company, e.g. the understandability and the clarity of

the interview questions to be used in the Delphi study.

In this way also the willingness of representative

experts to be interviewed in the Delphi sessions,

would be increased. In the preparation phase the

research team consisted of four persons, respectively

two scientific researchers and two experts from the

company, respectively a well-selected SAFe expert

and the director Quality Management of the

company.

In the second phase the Delphi method was

applied, a.o. because of its suitability to facilitate

distributed data collection in a company. Five experts

have been selected from different business units of

the company, respectively a Product Manager

Director, a Clinical Science Director, a Program

Manager, a Program Implementation Manager and a

Senior Product Manager. All experts were selected

because of their roles and tasks regarding the

communication with customers and their

involvement in SAFe projects in the company. Each

expert participates in Delphi sessions independently

from the other representatives. As such the Delphi

method eliminates undesirable group effects, such as

destructive dominance of a more powerful and

influential participant, and conformance pressure

within a group. Delphi refers to a structured process

with flexible iteration rounds with controlled

feedbacks aimed at obtaining reliable judgments and

opinions of a group of participants anonymously. In

our case study a two-round Delphi study was carried

out. The first Delphi round was used for individual

brainstorming on the customer involvement concepts

as identified in our previous reseach and as prepared

in the first phase. The second round was used for a

verification of the results of the first round, a

confrontation of the different results of the experts,

and for a justification of expert’s suggestions, all

based on controlled feedback. For each of the two

Delphi rounds, a well-elaborated set of questions was

defined in accordance with Delphi guidelines and

protocols.

Step 2: Extending the model (and linking findings

to the SAFe levels)

In this step information gained from the Delphi study

has been used to extend and enrich the initial

customer involvement concepts. Where possible,

links between the extended concepts and the SAFe

levels, see Table 1, have been suggested.

Step 3: Validation

In the validation step all the experts, two from the

preparation phase and five from the Delphi-based

phase, have been asked individually in a focused

validation interview a small number of questions

(Wieringa and Moralı, 2012). These questions were

of the type “will the extended conceptual model work

in the day-to-day practice of the company? Why?

And: will the model satisfy the goals of the users?

Why”? Subsequently the individual results were

collected, analysed and combined and the total result

was fed back to the respondents for validation.

4 RESULTS FROM THE CASE

STUDY

In this section the results of the three steps in our case

study will be presented and discussed. These results

cover both an improved understanding of the existing

concepts as well as the identification and definition of

new customer involvement (sub)concept in the

company. In the following first the preparation phase

of the data collection will be addressed briefly.

Subsequently the results of the diagnosis and analysis

of each of the identified (sub)concepts will be

discussed, in connection with validation aspects.

The preparation phase of the Delphi data

collection resulted in information about the way the

company deals with customer involvement in SAFe

projects. Information was obtained about the goals of

involving customers (i.e. improving product

acceptance and product quality), the stages in that

customers could be involved (i.e. to be decided by

project manager and product managers), and how to

involve customers in SAfe projects (i.e. via personal

invitations or scheduled company visits). In particular

a suitable terminology for implementing customer

involvement concepts in the company has been

identified. Concepts identified, partly redefined from

our initial study, are respectively: project risk

identification, customer involvement level, customer

product utilization and competence, and customer

motivation. Based on this information a two-round

Delphi study has been executed, of which the results

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

106

will be discussed in section 5.1. Section 5.2 will

address validation aspects.

4.1 Data Collection, Analysis And

Model Development

4.1.1 Project Risk Identification

The need for project risk identification, in relation to

customer involvement, was based on the need to

avoid or reduce project uncertainty. Uncertainty (and

risk) were caused by the lack of an approach (in

SAFe) to characterise a project in relation to customer

involvement. Clear concepts and terminology were

lacking as well as measures for characterising

projects. In accordance with (Applegate et al, 1996) it

was suggested to consider project risk as being

influenced by three project dimensions, respectively

technology, project size, and project structure. Based

on the usage of a 7-points scale (from a very low via

an average, to a very high risk), a poject typology has

been developed. The responding experts accepted to

handle this typology for their projects. Two

respondents stated that their projects were sometimes

of medium risk, and that as a consequence customer

involvement could be marginal, but that motivating

these customers should be a focus point. However,

four out of five respondents identified their projects

as often being of a very high risk, mainly because of

the high technology and the size of the projects. In

these cases customer involvement had to focus in

particular on the competence of customers to be

involved. Another discussion point was the

responsibility to involve customers. Should this be

done by project managers or product managers? In

the second round of the Delphi study it was

unanymously agreed that the reduction of risks

regarding customer involvement could be best

reached by making product managers responsible,

because of their close contacts with and knowledge of

customers. These kind of results showed that project

risk identification could be a promising step to reduce

uncertainty in the collaboration with customers.

However it was stated by all participants that extra

measures for project characterisation were needed to

make project risk identification operational.

Regarding the position of risk identification in the

SAFe framework (see table 1), it was suggested by

the respondents to position this on the Program and

Value stream level (in relation to e.g. project

management issues such as increment planning and

defining a roadmap).

4.1.2 Customer Involvement level

Regarding customer involvement level two sub-

concepts have been identified by three out of five of

the respondents in the first round of the Delphi study,

respectively customer position and customer

contribution.

Regarding the customer position, from a program

manager point of view, the customers are funders of

a project. When a customer is within the organization,

then the customer is identified as an internal

customer. Two of the responding experts defined

their customers as internal funders. However,

external customers, and e.g. a customer group, can

also be recognized as funders. These are customers

engaged in projects that build specific solutions, and

as mentioned by one of the respondents, in these

projects, customers are engaged deeply in the project,

and they have the authority to define project

requirements and specifications. In this case they

should be treated, and involved as ‘internal’

customers/funders, since they have a similar authority

as the customers who are from inside the

organization.The respondents agreed that it was most

common that the customer’s position was that of a

main project funder. Besides these ‘internal’

positions, general development projects were

mentioned by three out of five respondents, in that

customers have an external position.

Regarding customer contribution in product

development it appeared from the interviews that a

customer was authorised in two ways, respectively

for interpreting and deciding on project data, and on

contributing as a source of information. In literature

this is considered as contribution as a subject or as an

object (Holtzblatt and Beyer, 1993). Two of the five

responding experts allow involved customers to

interpret and decide on project data and to determine

the direction of a project (so contribution as a

subject). On the other hand three respondents stated

that their customers don’t have this authority.

Customers can only give input and feedback to a

product manager, so they act as an object . This

contribution can be very limited.

Figure 2 summarises the concept of customer

involvement level on the basis of position in the

organization and customer contribution. In general

solutions projects, i.e. with customers as objects, the

customer authority and contribution should be very

limited. I these projects, the product managers limit

customer involvement since requirements are

completely ‘frozen’ at the beginning of product

development.

Customer Involvement in the Scaled Agile Framework

107

Figure 2: Customer involvement level based on position

and contribution.

SAFe doesn’t recognize customers as subjects or

objects, but from the interviews it became clear that

projects in Philips use this subject and object concept

implicitly.

Determination of the customer involvement level

was suggested by the respondents to be positioned on

the Program and Value level as well as the Team level

of the SAFe framework. The position of a customer

can influence in particular the planning and control

of projects, while a particular cutomer contribution

can both influence verification and validation

activities (on the Program and Value level) as well as

the collaboration in development, e.g. the creation of

user stories (on the Team level).

4.1.3 Product Utilization and Customer

Competence

Regarding product utilization the business units in the

company appeared to make use of three types of so-

called ‘user persona’, implicitly or explicitly,

respectively first degree (direct use), second degree

(as provided by the direct user), and third degree user

(people who install, deploy or monetize the product).

This typology aims to give guidance to the project

manager to determine whose and what kind of needs

should be accommodated and prioritized. It appeared

that the first and the second degree user are the most

important customers (they directly make use the

product, and the company has to satisfy them).

Nevertheless, it also appeared that the third degree

user can be an important stakeholder because of their

authority to decide to invest in infrastructure or to buy

product components or not. Although the concept of

user persona is recognized in the Philips environment,

it is not applied explicitly, in a formal way, in the

business units.

Customers can be identified based on their

competence, e.g. ordinary users, experts, and lead

users (Magnusson, 2009). This specification helps the

product managers to discuss and identify the abilities

and knowledge of customers. The ordinary user is a

customer who understands how the features of the

product should be, but they don't understand the

technology that will be used. The experts understand

the technology that will be used, but they don't

understand how the features of the product should be.

The lead users are customers who understand the

features of the product as well as the technology to be

used. From the interviews it appeared to be difficult

to find lead users. In SAFe projects only ordinary

users and experts were involved. Product managers

make use of these distinct customer competences,

although it was stated that experts are not involved as

deeply as ordinary users.

The Delphi study revealed that the company has

implemented the concept of customer competence

implicitly. All respondents agree that user persona

and customer competence are relevant concepts to be

implemented in SAFe projects. Regarding the SAFe

framework, it was suggested by the respondents to

position product utilization and competence on the

Program level, because of the importance of product

utilization and competence of customers in evaluating

systems and participations in inspections.

4.1.4 Customer Motivation

A company has to ensure that involved customers

have a congruence of motivations and goals to

achieve a successful product development. SAFe

gives only limited information regarding how to

attract the customers. Therefore, we adopted a

concept formulated by (Fuller, 2010), who classifies

extrinsic customer involvement motives, which are

financial factors, social factors, and technological

factors. Intrinsic motivation is recognised, as a

condition when people enjoy to do something without

expecting a compensation or rewarding. In this study

intrinsic motivation is defined as consisting of

psychological factors.

It appeared in our case study that although the

mentioned concepts are relevant for the business

units, the responding experts couldn’t freely apply

them in their daily work. For example, in some

regions, the business units can’t offer financial

benefits to the customers because of local or

governmental regulations. A financial benefit, such as

the compensation or hospitality, can there be

identified as bribery. Therefore, the company focuses

in general at social, technological and psychological

factors to attract the customers for getting involved.

It appeared that the company also carries out

preventive actions regarding customers who feel to be

treated unfairly in case of different rewardings. From

the Delphi study it became clear that in SAFe projects

these motivation aspects are never addressed

explicitly. It was suggested to adopt these concepts at

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

108

the Value and Program level due to the close

interrelations with value issues.

Figure 3 shows the extended customer

involvement model. On the main level four concepts

are defined, respectively Project risk identification,

Determining the customer involvement level,

Customer specification, and Motivating the customer.

As can be seen two sub-concepts, i.e. Project risk and

involvement level play a role within more than one

main concept.

Figure 3: The extended customer involvement model.

5 VALIDATION

Regarding the validation of the work done we refer to

(Yin, 2013). We recognise four types of tests to

establish the quality of the research, which are

respectively construct validity, internal validity,

external validity and reliability.

Construct validity focuses on the justification and

correctness of the applied concepts and their

interrelations. Internal validity addresses the

understanding and the needs for new concepts

(customer involvement). In this study a preparation

phase for the data collection was executed with

feedback and control loops with well-selected highly

skilled company representatives. Customer

involvement concepts from literature were discussed,

eventually tuned to the characteristics of the company

(e.g. terminology), and questions for semi-structured

interviews in the Delphi study were developed. In the

preparation study and the two-round Delphi study a

chain of evidence has been established. In each round

of interviews feedback has been given, so answers

could be checked and eventually revised in case of

misinterpretations. Regarding the external validity we

can state that the resulting customer involvement

model has been established with information from

five experts from five different business units. So,

within the multinational company a particular level of

generalisation of the model has been established.

Reliability focuses on ensuring that the result of

the research is the same if the research should have

been conducted by other people (Yin, 2013).

According to this concern, this study performed a

formal and structured way of a preparation and a two-

round Delphi study with semi-structured interviews.

According to the input and feedback from the

respondents, the customer involvement model has

been extended and enriched. This model has then

been presented for discussion to the involved experts.

In this final session the model was validated by all the

involved experts, i.e. the SAFe agile expert, the

Quality Manager, and the five selected experts from

the business units.

6 CONCLUSIONS

This paper represents a partial achievement of a

longer-term project, i.e., the development of an

approach for customer involvement to improve SAFe

implementations. An initial conceptual model,

developed in previous research, was based on a

structured literature review and analysis. This paper

presented and discussed the extension andr

enrichment of the initial model. In our study we used

an inductive approach, i.e. carrying out semi-

structured interviews in the context of a two-round

Delphi study. Based on the results we presented the

extended customer involvement model. The

responding experts, two groups of respectively two

experts of the central organisation and five experts of

representative business units, have validated the

distinct elements of the model. In most cases the

(sub)concepts have been recognized in practice

although they were only applied in an incomplete

and/or implicit way. For each of the discussed

customer involvement concepts, suggestions have

been given to implement them at the distinct levels of

the SAFe framework.

REFERENCES

Abrahamsson P., Warsta J., Siponen M.T. and Ronkainen

J., 2003. “New directions on agile methods: a

comparative analysis”. In Proceedings on Software

Engineering. 25th International Conference, pp. 244-

254. IEEE.

Füller J., 2010. Refining virtual co-creation from a

consumer perspective. California management review,

52(2), 98-122.

Customer Involvement in the Scaled Agile Framework

109

Ghobadi S., and Mathiassen L., 2016. Perceived barriers to

effective knowledge sharing in agile software teams.

Information Systems Journal, vol. 26, no. 2, pp. 95-125.

Holtzblatt, K., and Beyer H., 1993. Making customer-

centered design work for teams. Communications of the

ACM, 36(10), 92-103.

Laage-Hellman J., Lind F., and Perna A., 2014. “Customer

involvement in product development: an industrial

network perspective”. Journal of Business-to-Business

Marketing, vol. 21, no. 4, pp. 257-276.

Laanti M., 2014. “Characteristics and Principles of Scaled

Agile”. In: International Conference on Agile Software

Development, Springer International Publishing, pp. 9-

20.

Leffingwell D., 2010. “Agile software requirements: lean

requirements practices for teams, programs, and the

enterprise”. Addison-Wesley Professional.

Magnusson P., 2009. “Exploring the Contributions of

Involving Ordinary Users in Ideation of Technology-

Based Services”. Journal of Product Innovation

Management, vol. 26, no. 5, pp. 578-593.

Mingers J., 2001. Combining IS research methods: towards

a pluralist methodology. Information systems research,

12(3), 240-259.

Nambisan S., 2002. “Designing Virtual Customer

Environments for New Product Development: Toward

a Theory”. Academy of Management Review, vol. 27,

no. 3, pp. 392-413.

Olson E. and Bakke G., 2001. “Implementing the lead user

method in a high technology firm: A longitudinal study

of intentions versus actions”. Journal of Product

Innovation Management, vol. 18, no. 6, pp. 388-395.

Sauvola T., 2015. “Towards customer-centric software

development: a multiple-case study”. In: Software

Engineering and Advanced Applications (SEAA), 41st

Euromicro Conference, IEEE, pp. 9-17.

Scaled Agile Framework, 2017. Retrieved from http://www

.scaledagileframework.com/, November.

Schwaber K. and Beedle M., 2002. Agile Software

Development with SCRUM. Prentice-Hall, 2002.

Trienekens J.J.M., Himawan H.B., van Moll J., 2017.

Customer Involvement in Scaled Agile Framework

Implementations, Towards a Conceptual Model as a

Basis for an Industrial Case Study, in: Proceedings of

the ACCSE, Venice, Italy, 2017.

Turetken O., Stojanov I., and Trienekens J.J.M., 2016.

"Assessing the adoption level of scaled agile

development: a maturity model for Scaled Agile

Framework." Journal of Software: Evolution and

Process, DOI: 10.1002/smr.1796, 2016.

Van Weele J., 2009. “Purchasing & supply chain

management: analysis, strategy, planning and

practice”. Cengage Learning EMEA.

Wieringa, R., and Moralı, A., 2012. Technical action

research as a validation method in information systems

design science. Design Science Research in

Information Systems. Advances in Theory and Practice

220-238.

Yin R.K., 2013. Case study research: Design and methods.

Sage publications.

ICEIS 2018 - 20th International Conference on Enterprise Information Systems

110