IT Project Portfolio Management: Development and Validation of a

Reference Model

Ruud Wissenburg, Rob Kusters and Harry Martin

Information Science Department, Open University, Valkenburgerweg 177, Heerlen, The Netherlands

Keywords: IT Project Portfolio, IT Project Portfolio Management, IT Project Portfolio Management Reference Model,

Portfolio Management Process, Portfolio Management Roles, Portfolio Management Responsibilities.

Abstract: IT Project Portfolio Management has been implemented in most organizations to effectively manage complex

portfolios of IT projects and balance them with business strategy. Several standards for portfolio management

have been published, but the scientific literature still lacks a theoretically grounded and practically validated

reference model for analyzing the implementation of IT Project Portfolio Management in an organization.

Therefore, this study designs and validates a reference model for systematically analyzing IT Project Portfolio

Management design choices in an organization in terms of processes, roles, responsibilities, and authority.

Organizations can use the reference model to systematically assess their local implementation of IT Project

Portfolio Management and identify areas for improvement.

1 INTRODUCTION

Organizations use projects to achieve their strategic

objectives and increase their competitive advantage

(Blomquist & Müller, 2006). Project portfolios are

important vehicles for achieving strategy as they

provide the link between strategy and operational

projects (Kock & Gemünden, 2021; Micán, Fernandes

& Araújo, 2022). Several studies have been conducted

on factors that determine the success of a project

portfolio (Hofman & Grela, 2015; Reyck et al., 2005;

Wissenburg, Kusters, Martin & Evers-Wagemakers,

2022). Nevertheless, many organizations struggle to

effectively manage and balance complex portfolios of

IT projects (Erasmus & Marnewick, 2020; Hoffmann,

Ahlemann & Reining, 2020). As a necessity, IT project

portfolio management (ITPPM) has been implemented

in most large organizations (Blomquist & Müller,

2006; Frey, 2014; Hoffmann et al., 2020). ITPPM

provides organizations with a holistic view of their IT

resources, should enable them to make more informed

IT-related decisions by identifying risks, benefits,

costs, and alignment IT resources (Ajjan, Kumar &

Subramaniam, 2016; Reyck et al., 2005). Thus, to be

successful, it is important for organizations to

adequately implement ITPPM.

Despite the fact that a global standard (PMI,

2017) and several portfolio management frameworks,

have been established, knowledge about the

successful implementation of ITPPM is limited so far.

Organizations still lack a reference to support the

development of their ITPPM governance (Ajjan et al.

2016; Kumar, Ajjan & Niu, 2008). The literature on

ITPPM lacks a theoretically based and practice-

validated reference model for ITPPM governance

(Ajjan et al., 2016; Hoffmann et al., 2020). This study

aims to establish such a model. We will demonstrate

the practical relevance of the reference model by

testing it against real-world ITPPM design choices.

The central research objective of this study is:

“Design and validate a reference model for ITPPM

governance, in terms of processes, roles,

responsibilities, and authority that can be used to

systematically analyze ITPPM design choices in

organizations.”

To achieve this objective, the literature was

searched for ITPPM processes, roles, and

responsibilities, and authority. A prototype reference

model was then designed based on the literature

studied. This prototype was validated with actual

ITPPM practices in a multiple-case study.

Organizations can use the reference model to

systematically describe their local design in terms of

processes, roles, responsibilities, and authority. Then

the differences with the reference model can be

identified and meaningfully discussed. By comparing

the local design of ITPPM with the reference model,

areas for improvement can be identified.

202

Wissenburg, R., Kusters, R. and Martin, H.

IT Project Portfolio Management: Development and Validation of a Reference Model.

DOI: 10.5220/0011959400003467

In Proceedings of the 25th International Conference on Enterprise Information Systems (ICEIS 2023) - Volume 1, pages 202-211

ISBN: 978-989-758-648-4; ISSN: 2184-4992

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

2 RELATED WORK

This section examines the availability of ITPPM

reference models in the literature.

Reyck et al. (2005) developed a PPM

implementation plan in three stages. In each stage,

certain PPM processes should be installed. However,

their framework lacks the roles, responsibilities, and

authority for ITPPM. Maizlish and Handler (2005)

described a step-by-step method for the design of IT

portfolio management. This method consists of eight

phases for building the ITPP. They also defined the

roles within PPM processes based on a stakeholder

analysis. Frey (2014) described four focus areas with

respect to ITPPM. For each focus area, governance

structures are applied that include interrelationships.

Frey (2014) also distinguished several roles,

responsibility, and authority within ITPPM. The

Project Management Institute (2017) standard

described knowledge areas and process groups

(Lima, Monteiro, Fernandes, and Machado, 2016).

Roles, responsibilities, and authority for portfolio

management are also described in this standard.

In summary, the recent literature contains a

(limited) set of elements of a reference model, but

lacks a complete and validated reference model. This

gap in research prompted this study with the central

research objective posed in the introduction.

3 METHODOLOGY DESIGN

This study aims to establish a practice-validated

reference model for systematically analyzing the

design of ITPPM governance in terms of processes,

roles, responsibilities, and authority. A so called

RACI matrix can be used to achieve this objective

because it reflects four types of process

responsibilities: the role who owns the problem

(responsible), the role of who must approve the work

before it is effective (accountable), the role that

provides input to help complete the task (consulted),

and people who need to be kept informed (informed)

(Cabanillas, Resinas & Ruiz-Cortés, 2012).

Therefore, the artifact of this study (i.e., the reference

model) will include a RACI matrix with process

activities, roles, responsibilities, and authority.

We wanted to get well-founded information from

practitioners based on their experience with the actual

design of ITPPM in their organizations. The design of

this study focuses on practice-based evidence (Aken

& Andriessen, 2011). Design science is an

appropriate research strategy for this purpose,

because it involves scientifically designing and

testing a solution to a practice problem through

systems and IT artifacts (Aken & Andriessen, 2011;

Hevner, March, Park & Ram, 2004). The cyclical

process followed in this study is based on design

science (Wieringa, 2014), consisting of the steps:

1) structured literature review;

2) designing the prototype reference model;

3) design-oriented case study;

4) reanalysis and additional structured literature

review (SLR);

5) redesign of the prototype reference model;

6) validation-oriented multiple-case study;

7) cross-case analysis;

8) redesign of the prototype reference model.

As described here, we designed a first prototype

based on the SLR, conducted an first validation in

practice, followed by a redesign and a final validation

in a multiple-case study. The choice for a case study

as a means of validation is justified by the richness of

the information that can be obtained. These research

steps are described in more detail below.

- Structured literature review (step 1):

To compose the components of a prototype reference

model, a s SLR was conducted to search the academic

literature for ITPPM processes, roles, responsibilities,

and authority. ITPPM is a widely studied topic in the

academic literature: therefore, exploratory research

methods such as the Delphi technique, the nominal

group technique, and focus groups were not

necessary. The SLR in this study was conducted

according to Kitchenham’s (2004) guidelines.

The purpose of our literature review was to

include a wide range of publications on ITPPM.

Therefore, we used the generic search terms “IT

project portfolio”, “portfolio management

processes”, “IT portfolio management step-by-step”,

“portfolio management roles”, and “project portfolio

management" process role responsibility

competence. The Open University’s digital library

was used to collect the scientific literature. This

provides access to IEEE, Elsevier, ACM, JSTOR,

Springer, and Web of Science, among others.

The articles found were screened based on a

number of inclusion criteria. Namely, the articles had

to be peer reviewed, written in the English language,

published after 2000, and have ITPPM as their topic.

The review process first checked whether the articles

had been published in peer-reviewed journals.

Second, the title and abstract were reviewed to

determine if the article was related to ITPPM. Articles

that did not address ITPPM were rejected. Third, the

remaining articles were scanned for ITPPM

processes, roles, responsibilities, and authority.

IT Project Portfolio Management: Development and Validation of a Reference Model

203

- Designing the prototype reference model (step 2):

The ITPPM process models found in the literature

were compared for completeness in terms of

processes, roles, responsibilities, and authority. Then

the most complete model was chosen and it was

determined what additions from other models were

needed. A RACI matrix was then created in which the

reference roles from the literature were placed on the

horizontal axis and all processes and activities on the

vertical axis. The boxes in the RACI matrix indicated

whether the roles involved in ITPPM were

responsible, accountable, consulted or informed for

the process activity in question.

This approach resulted in a prototype ITPPM

reference model in terms of processes, roles,

responsibilities, and authority.

- Design-oriented case study (step 3):

The purpose of this research step was to obtain

feedback on the quality of the prototype reference

model. To test the quality of the design, experiences

with ITPPM in the real context of an organization

were collected. A case study approach was used for

this strategy. The case organization had to include an

active ITPP, so it was plausible that they had practical

experience with ITPPM.

Testing the prototype reference model consisted

of five steps:

a) establishing the organizational model;

b) validating the organizational model;

c) comparing the organizational model with the

prototype reference model;

d) discussing the differences between the

organizational model and the reference model;

e) analyzing the interview data.

- Establishing the organizational model (3a):

The first step was to establish the organizational

model based on the organization’s, which resulted in

a first version of the case organization's RACI matrix.

- Validating the organizational model (3b):

To ensure that the established organizational model

could serve as a solid foundation for the interviews

with the practitioners, this model was first discussed

with an experienced portfolio manager from the case

organization. In the interview, the various

components of the reference model were discussed.

Inaccuracies or additions were then incorporated into

the organizational model.

- Comparing the organizational model with the

reference model (3c):

An analysis was then made of this organization-

specific model in relation to the prototype reference

model. The purpose of this analysis was to arrive at

some general statements that indicate notable

differences in the organization of ITPPM from a

theoretical reference model versus the practical

situation. This was accomplished by comparing the

two RACI matrices side by side and noting the

differences. This analysis resulted in a list of general

statements that was used to create the interview

protocol.

- Discussing the differences between the

organizational model and the reference model (3d):

To gather in-depth information about practitioners’

experiences and opinions about actual ITPPM

practices in an organization, interviews are useful

(Saunders et al., 2011; Wieringa, 2014; Yin, 2018).

Observation of ITPPM practices was not feasible

given the time available for this study. This is because

the ITPPM-cycle has a lead time of at least one year.

Interviews can be categorized into three types:

structured, semi-structured, and unstructured. Semi-

structured interviews allow for asking additional

questions and better interaction with respondents than

structured interviews (Saunders et al., 2011). In this

study, semi-structured face-to-face interviews were

conducted with practitioners in the field of ITPPM to

provide the opportunity to ask additional questions

about their actual experience with ITPPM in the

context of their organization and their arguments and

preferences for ITPPM design choices.

Consistent with the expertise requirements of

Skulmoski, Hartman, and Krahn (2007), respondents

had to have worked within the organization for at

least three years in a role with executive

responsibility within ITPPM. The group of

respondents had to include as much diversity of

different roles as possible. Multiple perspectives

ensure that ITPPM is viewed from more angles and

potentially yield more consistent findings. In

addition, sufficient time had to be allocated for each

interview.

The purpose of the interviews was to determine

whether the general statements about the differences

in the design of ITPPM within the case organization

compared to the prototype reference model were

recognized by the respondent and what the

respondent’s opinion on this was. Therefore, a

protocol was designed for the interview, consisting of

the following questions:

introduction;

2. verification that the general statements regarding

the differences in the design of ITPPM were

recognized by the respondent;

Why do you think this design was chosen within

your organization?

3. Respondent’s assessment of these differences;

Which design choice do you think is better for

the organization and why?

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

204

For each question, it was verified that the question

had been fully answered. If not, additional questions

were asked. The interviews were recorded and

transcribed. The transcripts were then returned to the

respondents for approval.

The result of the interviews was a clear picture of

the differences between the organization's ITPPM

model and the prototype reference model, along with

the respondents' arguments and preferences for

ITPPM design choices.

- Analyzing the interview data (3e):

The purpose of the data analysis was to determine the

respondents' arguments and preferences for the

differences in the design choices of the case

organization compared to the prototype reference

model. This can be analyzed by text analysis of the

transcripts of the interviews with the practitioners. By

extracting relevant words or (parts of) sentences from

the text that reflect these expressed arguments and

preferences, an overview can be created within the

data (Alholjailan, 2012). Another aim was to

categorize respondents' answers based on the same

types of arguments and preferences. A method

suitable for this purpose is the In vivo coding method

combined with axial coding. In vivo coding was used

to avoid researcher bias and axial coding was used to

relate codes to each other and categorize them with an

overarching description (Saunders et al., 2011).

- Reanalysis and additional structured literature

review (step 4):

To improve the quality of the prototype reference

model, a reanalysis was conducted independently by

a number of research team members to reduce

individual bias. First, a SLR was performed in the

same manner as described in step 1. The search terms

used can be requested from the corresponding author

of this article. Second, the results of the SLR were

compared to the initial reference model and

modifications were suggested by the research team.

- Redesign of the prototype reference model (step 5):

Upon completion of the analysis, the results of all the

analyses had to be merged into a single redesign of

the reference model. To reduce the biases of the

research team members during the analysis phase, a

joint design session was held. After the session, a

modified reference model was available and then

validated in multiple-case studies. Validation was

necessary given the scope of the redesign.

- Validation-oriented multiple-case study (step 6):

The purpose of this study is to develop a relevant

reference model for ITPPM governance, validated by

practical experiences and explanations of

practitioners. This information can be obtained in

organizations that employ ITPPM practitioners who

may have actual experiences with ITPPM. Moreover,

experiences and explanations of them were important

to discover whether our reference model is relevant to

support the development of ITPPM governance in an

organization’s real-world context. Therefore, a case

study approach was used as a validation strategy,

where all interviews within one organization were

considered as one case. Furthermore, within design

science, a case study approach is appropriate as a

validation method to examine an artifact (i.e. the

designed reference model) in an organizational

environment (Wieringa, 2014). The unit of analysis

was determined as the ITPPM processes of the case

organization. Therefore, a holistic design was chosen

over an embedded design.

Based on availability (Benbasat et al., 1987) and

the time constraints of the study and in accordance

with Eisenhardt's (1989) advice for the number of

cases in a multiple-case study, we planned four cases

in parallel. For the case organizations and

respondents, the same criteria applied as for research

step 3d. For the interviews, the same questionnaire

was used as in step 3. The interview transcripts were

analyzed in the same way as in research step 3e.

- Cross-case analysis (step 7):

Our multiple-case study involved collecting and

analyzing data from four case organizations. In a

multiple-case study, there are two stages of analysis -

the within-case analysis and the cross-case analysis

(Merriam, 2009). The within-case analysis was

conducted with the previous research step. Each case

was analyzed as a comprehensive case in itself. Once

the analysis of each case is completed, the cross-case

analysis began (Merriam, 2009). The purpose of the

cross-case analysis was to analyze the differences and

similarities between the reference model and the local

design of ITPPM in the four case organizations and

then to reach a conclusion about the validity of the

reference model. In addition, we wanted to determine

whether there were similarities among the

organizations in preferences for the reference model

or the organizational model. Comparison of the

results of within-case analyses will further enhance

the transferability of the study. For the cross-case

analysis, a table was designed that showed the

differences between the reference model and the

respective organizational models. Based on this table,

the similarities and differences between the case

organizations became apparent and conclusions could

be drawn about the validity of the reference model.

- Redesign of the prototype reference model (step 8):

The purpose of this research step was to determine

whether a redesign was needed based on the results of

the cross-case analysis.

IT Project Portfolio Management: Development and Validation of a Reference Model

205

In this study, the criteria credibility, transferability,

dependability, and confirmability were used to ensure

trustworthiness of our study (Guba,1981).

To promote credibility, perspectives were obtained

from multiple participants from different case

organizations (Maimbo & Pervan, 2005). In addition,

source triangulation was used through a documentary

study and interviews, which increased support for the

conclusions of our study (Benbasat et al., 1987). To

reduce researcher bias, the organizational model was

validated by a portfolio manager from the case

organization prior to the interviews. Moreover, the

process activities in the prototype reference model

were provided with definitions. Also, the interview

questions were based on the central research objective.

During the interviews, questions were clarified as

needed. Furthermore, the interviews were recorded and

transcribed. Subsequently, the transcripts were

returned to the respondent for approval. During the

analysis phase, in vivo coding was used to stay close to

respondents’ words. In addition, the case analyses was

reviewed by another researcher to reduce possible

researcher bias.

To promote transferability, information was

provided about the case organizations and

respondents who participated in this study, as well as

the methods used for data collection. Transferability

was also enhanced by repeating the analysis within

four case organizations and comparing them to the

other organizations in a cross-case analysis.

Research's dependability was increased by

defining and substantiating the research approach,

data collection and analysis, and conclusions. In

addition, the derivation of the conclusions from the

case data was made explicit.

Confirmability of the respondents' arguments and

preferences was increased by contributions from

respondents from different organizations.

Confirmability was also promoted by a description of

deficiencies in the research methods and their possible

effects, and an in-depth methodological description.

4 RESULTS

This section presents the results of the research steps.

- Structured literature review (step 1):

The number of articles found, reviewed and used per

search query can be requested from the corresponding

author of this article. Some of the literature on ITPPM

exists in the form of practitioner guidelines and

professional standards developed and disseminated

by leading organizations that have a significant

influence on the field (ul Musawir, Abd-Karim &

Mohd-Danuri, 2020). Therefore, this type of literature

was not excluded from selection in this study.

- Designing the prototype reference model (step 2):

The ITPPM process models found in the literature

were compared for completeness in terms of

processes, roles, responsibilities, and authority.

Maizlish and Handler's (2005) model was chosen as

the basis for the reference process. This model is

more comprehensive than the models in the other

articles and it is composed of detailed subprocesses

and activities, where activities can be assigned to

roles. Next, the process descriptions in the found

articles were compared with the descriptions of the

processes and activities of Maizlish and Handler

(2005) and in this way missing processes and

activities in the model of Maizlish and Handler

(2005) were searched for. Two components were

missing, namely the resource management process

and the activity manage strategic change (Lima et al.,

2016) as part of the process assessing execution. Both

components were added to obtain a complete ITPPM

process. For the purpose of the resource management

process, Pennypacker (2005) provided a well-fitting

classification of sub-activities.

The literature found was then searched for role

descriptions and descriptions of responsibilities and

authority within the processes. These descriptions

were collected and interpreted to arrive at a RACI

matrix. From the literature, an A (Accountable) and

R (Responsible) were then added to the RACI matrix

for each activity, with the exception of the

communicating process. For this process, it could not

be determined from the literature which role is

responsible and which role is accountable. It was

assumed that stakeholder identification is an activity

that occurs at a more strategic/tactical level and that

the portfolio review board (PRB) is accountable and

the portfolio manager responsible. Because

determining the message to be communicated and

actually communicating this message to stakeholders

occurs at the tactical/operational level of ITPPM, it

was assumed that the portfolio manager is both

accountable and responsible for the creating

communication packages and delivering

communication subprocesses. In addition, it was

assumed that how the responsibilities for the C and I

roles are assigned in an organization depends on local

conditions. Therefore, no additions were included for

the C and I roles based on the researchers'

interpretation. Based on the results of the SLR, a

RACI matrix was designed as a reference model.

- Design-oriented case study (step 3):

The case organization, an university in the

Netherlands, originated from a research team

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

206

members’ professional network based on availability

and within the specified requirements for a case

organization. Respondents with different roles in

ITPPM were approached to promote source

triangulation. The five respondents who participated

in the interviews were identified in cooperation with

the case organizations.

The case study was conducted in accordance with

the design. All interview transcripts were read in full

and the relevant text fragments were highlighted.

That is, the text fragments in which respondents

argued the design choices in the case organization and

their preferences. The same types of arguments were

then categorized by axial coding.

Respondents' preferences for ITPPM design

choices are shown in Table 1 Preferences for design

choices. This table describes, for the first two

differences identified, the design according to the

prototype reference model versus the case

organization’s design and the respondents'

preferences. The entire table with all identified

differences and the respondents’ preferences can be

requested from the corresponding author.

Table 1: Preferences for design choices.

Reference model

design

Organizational

model design

Preferences

Steering by PRB Steering by ICT

governance board

and domain-

ecific tables

1 x reference

model and 4 x

organizational

model

Established project

/ portfolio

management office

No established

project / portfolio

management office

5 x reference

model

To illustrate, the first difference is explained below,

supported by some interview quotes from

respondents.

PRB, as recognized in theory, is split in the case

organization into ICT Governance board and Tables.

The following arguments were given for this:

• Involvement of stakeholders (5 respondents):

“The table is primarily intended to create support

for change and to promote cooperation across

organizational divisions.”

• Quantity and diversity of topics (2 respondents):

“Due to the quantity and diversity of topics, the

choice was made to set up with ICT Governance

Board and tables.”

• Attention to project portfolio (1 respondent): “If

a choice would have been made for an ICT

Governance Board without tables, then the

portfolio and the projects would not (be able to)

receive the attention they deserve.”

• Attention to the long term (1 respondent): “With

the tables included, there is more of a longer term

view.”

In conclusion, the prototype reference model was

tested within a case organization and recognized as

such by the respondents. The organization deviated

from the reference model in a number of ways. The

respondents provided arguments for these deviations.

- Reanalysis and additional structured literature

review (step 4):

The reanalysis and additional structured literature

review were conducted independently by four

research team members, followed by a joint redesign

based on the analysis results. During the joint

redesign session, consensus was reached through

open and honest discussion.

- Redesign of the prototype reference model (step 5):

The redesign resulted in a number of modifications to

the prototype reference model, including:

• The role of project portfolio management office

was added.

• The authority for the subprocess creation was

transferred to the PRB instead of the portfolio

manager.

• In the balancing process activity, the

responsibility and authority for selecting and

approving changes was split.

• Process activity managing strategic change was

placed out of scope.

• Process activity allocating resources was split

into allocating IT resources and allocating

business resources.

In addition, the processes and roles were provided

with a description derived from the literature and the

reference model was translated into Dutch.

The redesigned reference model is shown in Table

2. The reference model, including definitions and

scientific literature sources, can be requested from the

corresponding author.

- Validation-oriented multiple-case study (step 6):

To validate the redesigned reference model, four case

organizations were available. These case organizations

originated from the research team members'

professional network and met the requirements

specified in research step 3. Respondents who

participated in the interviews were identified in

cooperation with the case organizations based on the

roles in the RACI matrix (i.e., purposive sampling).

The purposive sampling technique means that a

participant is chosen because of the unique and rich

information the participant possesses that is of value of

the study (Etikan, Musa & Alkassim, 2016). The

validation of the reference model consisted of the five

steps described in research step 3 and were carried out

in accordance with the methodology design.

IT Project Portfolio Management: Development and Validation of a Reference Model

207

Table 2: Redesigned reference model.

Process activities / Roles

PRB

Portfolio Manager

Program Manager

Senior Managers

Customer

Sponsors

Project Managers

PPM Office

PM Office

Project Team

Line Managers

Architects

Finance Manager

Business Partners

Portfolio mana

ement

olic

R A

Business

rocess inte

ration R A

Governance com

liance tri

ers R A

Governance

rocessses R A

Establishing organization R A

Readiness assessment R A

Validation and refinement assessment R A

Maturit

assessment R A

anal

s + ca

abilit

assess. R A

Defining objectives R A C

Defining

rocess metrics R A

Document Im

lementation

lan R A

Plan investment strate

AR R

Plan

ortfolio structure A R

Plan individ. Sub

ortfolio’s A R

Populating portfolio A R C C C C

Identify expected result + risks A R C C C C

Definin

ect metrics A R C C

Buildin

ortfolio view A R

Finalisin

ortfolio A R

Monitoring triggers A R

Measuring portfolio A R

Com

arin

measures - tar

et AR

Identif

refinement o

tions AR C

Determinin

trade-offs AR C

Selecting / approving changes A R C C

Implementing changes AR

Assessing implementation

lan execution ARC CC C

Com

arison of

erformance assessment A R R

ort assessment A R R

Resource

ool

ana

ement

Allocating IT resources AR R R

Allocating business resources AR

Identif

stakeholders A R

Creatin

communication

ack.

acka

e AR

Deliverin

communication AR

Nineteen semi-structured interviews were

conducted with respondents in the four case studies.

The research team members analyzed all interview

transcripts from their case organization. That is, the

text excerpts in which respondents argued the design

choices in the case organization and preferences were

highlighted. Then, the arguments were categorized.

To illustrate, the first two differences in case

organization 1 are explained below supported by

some interview quotes from respondents.

Steering by PRB (reference model) versus

steering by a table structure (organizational model)

Four out of five respondents pointed to workload as the

reason for the existence of different tables. Pragmatic

reasons were also mentioned. The chair of the intake

table explained, "Otherwise it becomes very

complicated which people to have at such a table."

Other possible reasons mentioned were the required

knowledge of the respondents, alignment with chain

goals and decision making at different levels.

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

208

Financial manager (reference model) versus no

financial manager in the chain structure

(organizational model)

The lack of a financial manager in the chain has two

primary causes according to respondents. The

controller, "For the chain, it is more about setting

substantive priorities within the framework than real

financial management." In addition, the context of a

government organization was also mentioned as a

cause. The architect articulated this as "In a

commercial organization, if you don't get financial

management right, you immediately have a potential

business risk. For a government agency, this is a

different situation. "

In summary, the prototype reference model was

recognized within the four case organizations by the

respondents and thus validated in practice. The

organizations deviated from the reference model at a

number of points and the respondents provided

credible arguments for these deviating design

choices. Moreover, it became apparent during the

interviews that the reference model served well as an

instrument for systematically analyzing the current

design of ITPPM governance in the respective

organization. A total overview of the explanations for

the differences and their substantiation with interview

quotes can be requested from the corresponding

author of this article.

- Cross-case analysis (step 7):

During the cross-case analysis, the differences and

similarities were analyzed between the prototype

reference model and the local design of ITPPM in the

four case organizations. This revealed some

corresponding differences.

Firstly, the ITPPM maturity assessment and gap

analysis and capability assessment subprocesses were

not set up in two case organizations. However, eight of

the nine respondents expressed a preference for setting

up these subprocesses in their organizations. Secondly,

in the reference model, the PRB has authority for a

number of activities, while in three case organizations,

senior management has authority for these activities. A

possible explanation for this is what a respondent

reported, “This design has to do with the way the

organization prefers to organize decision-making,

which is along the lines of management teams and they

don't want to be taken by surprise''. Thirdly, in the

reference model, the PRB is responsible for the

governance and organization and implementation plan

subprocesses. In two case organizations, this

responsibility lies with individual functions, such as

the portfolio manager. A possible explanation for this

is what a respondent mentioned, “Within the

organization, we describe things focused on individual

officers.” Fourthly, according to the reference model, a

limited number of roles are consulted during process

activities. In two case organizations, however, more

extensive role consultations takes place. A possible

explanation for this is what a respondent reported,

"Sometimes too many roles gets a change to say

something. This fits the organization's culture."

Another respondent pointed out that this is related to

the complexity of the organization. Lastly, the

reference model does not include roles for the process

activity resource pool management. Two organizations

did assign roles to this process activity. This can be

explained by the lack of an adequate role description of

this activity in the literature.

- Redesign of the prototype reference model (step 8):

The cross-case analysis revealed limited

corresponding differences between the prototype

reference model and the local design of ITPPM in the

four case organizations. We argue, therefore, that the

reference model is of sufficient quality and has been

validated in this way. A redesign of the reference

model was not necessary for that reason.

5 CONCLUSIONS

The research approach and research results are

discussed in this section. Conclusions are also drawn

and recommendations made for the use of the

reference model in practice and future research.

This study first conducted a SLR, which resulted

in an overview of ITPPM processes, roles,

responsibilities and authority, and their descriptions.

Second, a RACI matrix was created based on these

components. Third, this prototype reference model

was tested in a single case study. Fourth, the reference

model was reanalyzed and redesigned, and fifth, this

redesigned prototype reference model was validated

in a multiple-case study by the experiences of ITPPM

practitioners. Finally, the practitioners’ arguments for

local design choices were categorized and listed.

These research steps were conducted according to the

methodology design and proved suitable for arriving

at a validated reference model for ITPPM governance

in terms of processes, roles, responsibilities, and

authority, so that redesign was not necessary.

Recent literature lacks a comprehensive, practice-

validated reference model for ITPPM governance.

Therefore, this study has provided a comprehensive

and empirically validated reference model for ITPPM

governance in terms of processes, roles,

responsibilities, and authority. In addition, local

deviations from the reference model are indicated.

Practitioners were able to provide credible evidence

IT Project Portfolio Management: Development and Validation of a Reference Model

209

for these deviations. These deviations can be

considered local practices and may be applicable to

other organizations as well. Moreover, during the

multiple-case study, our model proved to be an

appropriate reference to systematically analyze and

discuss past design choices in an organization, and

identify areas for improvement. It prompted

respondents to reflect on past ITPPM design choices

within the organization. Therefore, we argue that the

research findings of this study provide a strong

indication of the validity of the reference model.

Thus, we achieved the objective of this study.

A first limitation of this study is that it is based on

respondents' recollection and perceptions of their

experiences with ITPPM rather than on our own

direct observations in the field. A second limitation is

that it is based on the quality of practitioners'

experiences. Proper selection of respondents is

important in this regard. Therefore, we used

respondents’ experience as a selection criterion.

Respondents had to have worked within the

organization for at least three years in a role related to

ITPPM. However, experience does not necessarily

guarantee expertise. Sengupta, Abdel-Hamid, and

Van Wassenhove (2008) pointed out that experts do

not always learn from their experience effectively.

Learning in practice means that actions and outcomes

are systematically evaluated, which practitioners very

often have difficulty to find time for. We were not

able to test the concrete expertise of the respondents

beforehand. A third limitation is that pragmatic

validity (Aken & Adriessen, 2011) was not tested.

Practitioners evaluated the prototype reference model

based on their actual experiences, but the theoretical

reference model was not used for the actual design of

ITPPM in organizations.

Some interesting deviations were found during

the case studies, where respondents substantiated

their preference for local design in their

organizations. Therefore, it can be concluded that our

model is a good reference, but local conditions should

be taken into account during development of ITPPM

governance. There is no one size fits all approach,

rather ITPPM processes and roles needs to be tailored

to best fit an organization and its context (Blomquist

& Müller, 2006; Castro & Carvalho, 2010; Killen &

Hunt, 2011; Kock & Gemünden, 2021).

Academic research to date provides limited

insight into the organizational factors that influence

ITPPM implementation (Ajjan et al., 2016). Future

research could focus on the factors that contribute to

successful ITPPM implementation. Furthermore,

organizations with mature ITPPM are more

successful than those with less mature ITPPM

(Jeffery & Leliveld, 2004; Mosavi, 2014; Zarghami

& Dumrak, 2020). These differences in maturity

leading to different levels of benefits highlight the

importance of figuring out which ITPPM processes

are needed to deliver higher levels of benefits to the

organization.

ACKNOWLEDGEMENTS

We would like to thank all respondents for their

valuable opinions and contributions to the project.

They have set aside the necessary time for the

interviews. Special thanks to the following research

team members for their field work: Gérard de Smaele,

Willem Post, Nienke Blauw, Lars Driessen, and

Jamal Ibrar Hussain, Open University. We are also

indebted to the anonymous reviewers for their helpful

comments.

REFERENCES

Ajjan, H., Kumar Ram, L. & Subramaniam, C. (2016).

Information technology portfolio management

implementation: a case study. Journal of Enterprise

Information Management, 29(6), 841-859.

Aken, J. van & Andriessen, A. (Eds.). (2011). Handboek

ontwerpgericht wetenschappelijk onderzoek. Den Haag,

The Netherlands: Boom Lemma.

Alholjailan, M.I. (2012). Thematic Analysis: A critical

review of its process and evaluation. West East Journal

of Social Sciences, 1(1), 39-47.

Benbasat, I., Goldstein, D. & Mead, M. (1987). The case

research strategy in studies of information systems. MIS

Quarterly, 11(3), 369–386.

Blomquist, T. & Müller, R. (2006). Practices, Roles, and

Responsibilities of Middle Managers in Program and

Portfolio Management. Project Management Journal,

37(1), 52-66.

Cabanillas, C., Resinas, M., & Ruiz-Cortés, A. (2012,

September). Automated resource assignment in BPMN

models using RACI matrices. In OTM Confederated

International Conferences "On the Move to Meaningful

Internet Systems" (pp. 56-73). Springer, Berlin,

Heidelberg.

Castro, H. G. D. & Carvalho, M. M. D. (2010).

Gerenciamento do portfólio de projetos (PPM): estudos

de caso. Production, 20(3), 303-321.

Cooper, R. G. & Edgett, S. J. (2001). Portfolio Management

for new Products: Picking the winners. Ontario, Canada:

Product Development Institute.

Cooper, R. G., Edgett, S. J. & Kleinschmidt, E. J. (1997).

Portfolio management in new product development:

lessons from the leaders – II. Research Technology

Management, 40(6), 43-52.

Eisenhardt, K. M. (1989). Building theories from case study

ICEIS 2023 - 25th International Conference on Enterprise Information Systems

210

research. Academy of management review, 14(4), 532-

550.

Etikan, I., Musa, S. A., & Alkassim, R. S. (2016).

Comparison of convenience sampling and purposive

sampling. American journal of theoretical and applied

statistics, 5(1), 1-4.

Erasmus, W. & Marnewick, C. (2020). An IT governance

framework for IS portfolio management. International

Journal of Managing Projects in Business, 14(3), 721-

742.

Frey, T. (2014). Governance Arrangements for IT Project

Portfolio Management: Qualitative Insights and a

Quantitative Modelling Approach. Wiesbaden,

Germany: Springer.

Guba, E. G. (1981). Criteria for assessing the trustworthiness

of naturalistic inquiries. Educational Communication

and Technology Journal, 29(2), 75–91.

Hevner, A. R., March, S. T., Park, J. & Ram, S. (2004).

Design Science in Information Systems Research. MIS

Quarterly, 28(1), 75-105.

Hoffmann, D., Ahlemann, F. & Reining, S. (2020).

Reconciling alignment, efficiency, and agility in IT

project portfolio management: Recommendations based

on a revelatory case study. International Journal of

Project Management, 38(2), 124-136.

Hofman, M., & Grela, G. (2015). Project portfolio risk

identification-application of Delphi method. Journal of

Business and Economics, 6(11), 1857-1867.

Jeffery, M. & Leliveld, I. (2004). Best Practices in IT

Portfolio Management. MIT Sloan Management review,

45(3), 41-49.

Killen, C. P. & Hunt, R. (2009). Project portfolio manage-

ment maturity model for dynamic environments. In

Australian Institute of Project Management conference.

The Australian Institute of Project Management.

Killen, C. P., Jugdev, K., Drouin, N. & Petit, Y. (2012).

Advancing project and portfolio management research:

applying strategic management theories. International

Journal of Project Management, 30(5), 525-538.

Kitchenham, B. (2004). Procedures for performing

systematic reviews. Keele, UK, Keele University.

Kock, A. & Gemünden, H. G. (2021). How entrepreneurial

orientation can leverage innovation project portfolio

management. R&D Management, 51(1), 40-56.

Kopmann, J., Kock, A., Killen, C. P. & Gemünden, H. G.

(2017). The role of project portfolio management in

fostering both deliberate and emergent strategy.

International Journal of Project Management, 35(4),

557-570.

Kumar, R., Ajjan, H. & Niu, Y. (2008). Information

Technology Portfolio Management: Literature Review,

Framework, and Research Issues. Information Resources

Management Journal, 21(3), 64-87.

Lima, A., Monteiro, P., Fernandes, G. & Machado, R. J.

(2016, September). Mapping between artefacts and

portfolio processes from the PMI standard for portfolio

management. In EuroSymposium on Systems Analysis

and Design (pp. 117-130). Springer, Cham.

Maimbo, H. & Pervan, G. (2005). Designing a case study

protocol for application in IS research. Proceedings of

the Ninth Pacific Asia Conference on Information

Systems (PACIS 2005), 1281-1292.

Maizlish, B. & Handler, R. (2005). IT Portfolio management

Step-By-Step: Unlocking the business value of

technology. Hoboken, NJ, USA: Wiley.

Martinsuo, M. (2013). Project portfolio management in

practice and in context. International Journal of Project

Management

, 31(6), 794-803.

Merriam, S.B. (2009). Qualitative Research. A Guide to

Design and Implementation (2nd ed.). San Francisco,

Jossey-Bass.

Meskendahl, S. (2010) The influence of business strategy on

project portfolio management and its success – a

conceptual framework. International Journal of Project

Management, 28(8), 807–817.

Micán, C., Fernandes, G. & Araújo, M. (2022). A method for

project portfolio risk assessment considering risk

interdependencies–a network perspective. Procedia

Computer Science, 196, 948-955.

Mosavi, A. (2014). Exploring the roles of portfolio steering

committees in project portfolio governance. Internat-

ional Journal of Project Management, 32(3), 388-399.

ul Musawir, A., Abd-Karim, S. B. & Mohd-Danuri, M. S.

(2020). Project governance and its role in enabling

organizational strategy implementation: A systematic

literature review. International Journal of Project

Management, 38(1), 1-16.

Pennypacker, J.S. (2005). Project portfolio management

maturity model. Pennsylvania, USA: Center for Business

Practices.

PMI. (2017). A guide to the project management body of

knowledge (6th ed.). Pennsylvania, USA: Project

Management Institute.

Reyck, de B., Grushka-Cockayne, Y., Lockett, M., Calderini,

S. R., Moura, M. & Sloper, A. (2005). The impact of

project portfolio management on information technology

projects. International Journal of Project Management,

23(7), 524-537.

Saunders, M., Lewis, P., Thornhill, A., Booij, M. &

Verckens, J. P. (2011). Methoden en technieken van

onderzoek (M. Booij & J.P. Verckens, Trans. 5e ed.).

Amsterdam, The Netherlands: Pearson Education.

Sengupta, K., Abdel-Hamid, T. K., & Van Wassenhove, L.

N. (2008). The experience trap. Harvard Business

Review, 86(2), 94-101.

Skulmoski, G. J., Hartman, F. T. & Krahn, J. (2007). The

Delphi method for graduate research. Journal of

information technology education, 6, 1-21.

Wieringa, R. J. (2014). Design Science Methodology for

Information Systems and Software Engineering. London,

England: Springer.

Wissenburg, R., Kusters, R., Martin, H., & Evers-

Wagemakers, J. (2022, June). IT Project Portfolio

Assessment criteria: development and validation of a

reference model. In 2022 IEEE 24th Conference on

Business Informatics (CBI) (Vol. 1, pp. 136-145). IEEE.

Yin, R. K. (2018). Case Study Research and Applications:

Design and methods (6th ed.). Thousand Oaks, CA,

USA: Sage.

Zarghami, S. A., & Dumrak, J. (2020). Application of system

dynamics in the assessment of project portfolio

performance. International Journal of Industrial

Engineering and Management, 11(4), 253-262.

IT Project Portfolio Management: Development and Validation of a Reference Model

211