ITO-TRACKER

A Tool for Evaluating ITO Projects based on Critical Success Factors

Edumilis Méndez, María Pérez, Luis E. Mendoza and Maryoly Ortega

Processes and Systems Department – LISI, Universidad Simón Bolívar, Caracas, Venezuela

Keywords: Information Technology Outsourcing, Project Evaluation, Critical Success Factors, Metrics, SMEs.

Abstract: Nowadays, the Latin American software industry, as it is mostly represented by Small and Medium

Enterprises (SMEs), should focus on improving its service capacity towards high quality, low costs, and

timely delivery. Within this context, SMEs providing Information Technology Outsourcing (ITO) services

require information that allows to assess and monitor their contractual relationships and the agreements

contained therein, considering a set of critical success factors that may vary depending on the type of project

addressed. This article is aimed at describing the development of a tool named ITO-Tracker designed for the

evaluation of ITO projects oriented to software, hardware, network and databases, based on Critical Success

Factors (CSFs). ITO-Tracker offers guidance to the parties involved in the service through regular control

and follow-up activities. The methodology used for developing this tool was the iterative and incremental

process known as Rational Unified Process (RUP). For this purpose, we documented the development

process and results from ITO-Tracker conceptualization, analysis, design and construction. Besides it is

presented the evaluation of an ITO project in the software development area using the tool.

1 INTRODUCTION

The Latin American software industry is mainly

comprised of Small and Medium Enterprises

(SMEs) and it is still conceived as incipient and

immature (Mayer & Bunge, 2004). This has led to a

lack of competitiveness, which has impeded its

growth. However, the rise of Information

Technology (IT) and the abundance of software

development tools have made available valuable

tools to SMEs, not only to improve software

production, but also to evaluate the development

process or the product obtained (Rivas et al., 2007).

In addition, Information Technology Outsourcing

(ITO) is an increasingly widespread practice among

enterprises (Lee et al., 2003) and one of the most

prevailing businesses among SMEs, since these

enterprises are involved in the development,

maintenance and marketing of software products for

third parties (Rivero et al., 2007), mostly acting as

suppliers and sometimes as clients, and sometimes

they subcontract other SMEs to perform specific

tasks. Therefore, it is essential counting on tools that

allow evaluating those aspects that influence

successful relationships between the client and the

ITO provider; both wish to know their level of

success to control and monitor the status of their

projects, so that this information is used for their

enhancement.

This article describes the development of the

ITO-Tracker tool designed for assessing CSFs

proposed in a previous research (Méndez et al.,

2007a) and it is intended to provide SMEs with a

supporting tool for ITO projects’ evaluation. Also, it

provides guidance as to this service to the parties

involved, so that they can consolidate their strengths

and overcome their weaknesses for ITO projects’

development.

2 BACKGROUND

According to Fairchild (2004) and Sheehy et al.

(2003), ITO is defined as the organization’s practice,

which consists in contracting an external provider

with expertise in performing one or more IT

functions, so the enterprise does not have to do it

itself. In order to achieve a successful ITO, certain

practices known as CSFs should be considered.

Austin (2002) defines CSFs as critical areas where

satisfactory performance is required for the

organization in order to achieve its goals. For this

170

Méndez E., Pérez M., E. Mendoza L. and Ortega M. (2008).

ITO-TRACKER - A Tool for Evaluating ITO Projects based on Critical Success Factors.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - ISAS, pages 170-175

DOI: 10.5220/0001686701700175

 SciTePress

purpose, efforts in this research are directed towards

developing a tool supporting the CSFs model

associated to IT: data center, network, software

development and hardware support (O'Brien, 2005).

This model consists of 22 CSFs and 400 metrics

(Méndez et al., 2007a). Table 1 shows the model’s

CSFs used as a basis for defining the ITO-Tracker

requirements.

Table 1: Technological Critical Success Factors for ITO.

Critical Success Factors

1. Define services from a modular perspective

2. Agree on the transfer of initial resources

3. Agree on the ownership of new resources

4. Define a service evaluation structure

5. Define a predictable cost structure

6. Coordinate and standardize tasks integration

7. Invest in technology innovations and planning

8. Maintain ownership and internal capabilities

9. Consider licenses restrictions

10. Invest in value-added ecosystems, which integrate

services from different providers

11. Establish multilateral agreements with the right

providers

12. Consider corporate regulations

13. Consider governmental regulations

14. Consider personnel resources planning

15. Establish exclusivity agreements for key areas

16. Manage risks and assign responsibilities

17. Consider potential service changes

18. Define contract termination strategies

19. Accelerate services life cycles

20. Take into account cultural compatibility

21. Study client-provider relations

22. Learn from the experience of allies organizations

It should be noted that there is another tool used

for ITO project evaluation (Méndez et al., 2007b),

which is based on a CSFs model that focuses only

on Management aspects of the client-provider

relationship (27 out of 29 CSFs correspond to this

area), thus leaving aside technological aspects (only

2 out of 29 CSFs correspond to this area).

The tool developed for this work provides

support to researchers and SMEs in ITO projects

evaluation, since it considers technological aspects

and allows overcoming the former tool’s

weaknesses.

3 ITO-TRACKER

DEVELOPMENT

For developing this tool, the Rational Unified

Process -RUP- (Kruchten, 2003) methodology was

used. Due to space limitations, we will cover only

the most relevant aspects of the artifacts designed

for ITO-Tracker development.

3.1 Inception Phase

In this phase, it was collected information to

establish the needs, characteristics, requirements and

scope of the tool. This process was documented in

the Vision artifact.

The software requirements were identified and

documented in the Software Requirements

Specification (SRS) artifact. The functional

requirements are listed in Table 2.

Table 2: Functional Requirements of ITO-Tracker.

As to non-functional requirements, the system

should:

a) be user-friendly and include an easy-to-use

interface with menus and help options.

b) be available for a minimum of 40 hours

weekly (i.e. 8 business hours a day).

c) be fault-tolerant.

d) be able to provide a maximum 3-second

response and processing time

Functional Requirements

Log system users, Authenticate users

Create, Edit, Delete, View project types

Create, Edit, Delete, View, Assign/Remove CSFs

Create, Edit, Delete, View, Assign/Remove questions

Create, Edit, Delete, View, Assign/Remove question

metrics

Create, Edit, Delete, View points

Create, Edit, Delete, View enterprises

View enterprise’s projects

View enterprise’s users

View user profile

View user’s projects

View user’s enterprises

Create, edit, delete, view projects

View project’s users

Assign/remove user

View project’s enterprises

Evaluate projects

Generate Corporate Report on project Evaluations

Generate Research project on performance of CSFs

for similar projects between one enterprise and others

View projects (Project Leader)

View operations log for a project

ITO-TRACKER - A Tool for Evaluating ITO Projects based on Critical Success Factors

171

e) support approximately 50 concurrent users

f) be documented and allow for easy

maintenance.

g) consider access restrictions and encryption

devices for users’ logins.

Then, 67 use cases were identified for system

development. Figure 1 shows the general use cases

diagram, grouped by packages.

Figure 1: Use Case Model.

3.2 Elaboration Phase

The milestone of this phase is establishing the

system architecture that guarantees efforts invested

in the next phase’s design and implementation, the

main artifact being the Document of Architecture

Work. For system development, the Model-View-

Controller (Veit and Herrmann, 2003) architectonic

pattern was used which consists of three layers.

The architectonic views suggested are as follows

Use-Case View (Figure 1), Logical View (Figure 2),

Deployment View, Implementation View and Data

View. Due to space limitations, we will show only

the most relevant views.

Evaluation Question

Data Center

Hardware

Support

Network

Software

Development

Enterprise CSFProject

User MetricView Point

Researcher Specialist

Evaluation

Leader

***

indicates score

records score

has ^

contains

determines successmanages

belongs to ^

performs

is assigned

corresponds to

Figure 2: Domain model associated to the ITO-Tracker

system.

For Deployment View, it was decided that the

tool worked under a client-server architecture. The

application and database server is a Linux server

where the application is installed. Clients can use

any device with a web browser and can connect to

the server through local area network (LAN).

The Implementation View (Figure 3) shows the

components diagram with the main physical devices

included in the software deployment unit.

Figure 3: Hierarchy of ITO-Tracker System Components.

3.3 Construction Phase

The milestone of this phase consists in refining some

of the requirement specifications and

complementing the system development on its basic

architecture. Accordingly, the necessary

development environment was installed and all

product functionalities were constructed. All

architectural layers, the group of use cases proposed,

and the system database were implemented. In

addition, unit tests were applied to each use case.

This phases’ main deliverable was an ITO-Tracker

version, including the source code of all

implemented classes.

ITO-Tracker has been basically divided into four

sub-systems, as follows:

 Model Management: It includes management

of GQM-related objects (Basili et al., 1994)

used for operationalization of the model

(Figure 4).

 Evaluation Management: It includes object

management related with project evaluation.

 ITO Analysis: It corresponds to graphic

reports’ generation and view of results of the

ITO projects’ evaluations (Figure 5).

ICEIS 2008 - International Conference on Enterprise Information Systems

172

Researcher

Model Management

Evaluation Management

ITO Analysis

Administration

ITO Types Management

CSFs Management

View Points Management

Question Management

Metrics Management

Enterprises Management

Users Management

Projects Management

Reports

Operations Log

Profile

Exit

0001 Define services from a modular perspective Project leader

0002 Agree on the transfer of initial resources Project leader

0003 Agree on the ownership of new resources Project leader

0004 Define a service evaluation structure Responsible for Financial

0005 Define a predictable cost structure Project leader

0006 Coordinate and standardize tasks integration Project leader

ID View Point

Description Options

Be Located

Keyword

View point

Figure 4: ITO-Tracker Management Model Screen.

Researcher

Model Management

Evaluation Management

ITO Analysis

Administration

ITO Types Management

CSFs Management

View Points Management

Question Ma nagement

MetricsManagement

Enterprises Management

Users Management

Projects Ma nagement

Reports

Operations Log

Profile

Exit

0001 Define services from a modular perspec tive Project leader

0002 Agree on the transfer of initi al resources Project leader

0003 Agree on the ownership of new resources Project leader

0004 Define a service evaluation structure Responsible for Financial

0005 Define a predictable cost structure Project leader

ID View Point

Description Options

Be Located

Keyword

View point

Enterp rise

Reference

Project

SIGMSUAF

SIGMSUAF project´s evaluation

Date (Begining) Date (completion) Score Options

1 Generatedby engine 27/08/2007 03/09/2007 37.45%

Match with

lowercase l etters

SI GM SUAF evaluat ion

100

16,67

28,57

41,67

33,33

66,67

100

66,67

35,42

100

33,33

33,33 33,33

100

12345678910111213141516171819202122

CSF

Values

Figure 5: ITO-Tracker Analysis Screen.

 System Administration: It contains the

operations log that records all users’ activities.

3.4 Transition Phase

This phase is focused on the description of tasks

performed to guarantee availability of the

implemented ITO-Tracker to its final users.

Two separate installations were performed, first

at the enterprise owner of the project subject to be

evaluated; this installation allowed to establish a

project evaluation case study with automated

support; and at same time allowed experts involved

in the project pilot to make their observations, and

detect potential failures or improvements. The

second installation represented the tool’s transition

to the server of the Information Systems Research

Lab (LISI, by its abbreviations in Spanish) at Simón

Bolívar University (USB by its abbreviations in

Spanish), i.e. its final certification/pre-production

environment.

4 PROJECT EVALUATION AND

RESULTS

ITO-Tracker was used to evaluate a software

development project at the banking sector, named

SIGMSUAF. It provides a solution for prevention

and reduction of bank frauds, capable of monitoring

bank transactions in real time and detecting irregular

consumption patterns. Hope you find the

information in this template useful in the preparation

of your submission.

The software provider is in charge of configuring

the project evaluation with the tool, and preparing

the necessary environment to be used by evaluators;

these will receive an e-mail with their respective

the tool.

The project selected was incorporated as a

software development ITO. Roles considered for the

project evaluation included the following: Project

leader, Responsible for Financial Evaluation &

Purchases, Responsible for Technology Planning,

Responsible for Negotiations with Suppliers,

Responsible for Strategic Planning. Individuals

responsible for playing such roles were the

ITO-Tracker users.

ITO-Tracker displays for users/evaluators those

questions corresponding to each type of project. By

means of this tool, answers were provided to the

questions of the CSFs corresponding to each role

assigned during project development. Once the

evaluation was completed, the project leader queried

the results through different views provided by the

tool.

Figure 6 generated by ITO-Tracker tool shows

the detail of the SIGMSUAF evaluation per each

CSF. The evaluation results are as follows:

SIGM SUAF evaluation

100

16,67

28,57

41,67

33,33

66,67

100

66,67

35,42

100

33,33

33,33 33,33

100

12345678910111213141516171819202122

CSF

Values

Figure 6: Results of the pilot project evaluation.

Only 3 CSFs were 100% satisfied for the pilot

project. It can be observed from these results, that

the client is concerned about specification activities,

rights on new applications, and visualization of the

support of the personnel involved. Even though the

evaluation shows an efficient communication in the

project, we may infer it is only for specific stages or

activities, instead of for the entire project.

ITO-TRACKER - A Tool for Evaluating ITO Projects based on Critical Success Factors

173

The pilot project shows 2 CSFs that almost

reached a 75% satisfaction (CSFs 14 and 17) i.e. the

minimum percentage for CSF satisfaction; 2 CSFs

with a 50% satisfaction (CSFs 13 and 19), and 10

CSFs between 0% and 49% (CSFs 1, 3, 4, 5, 6, 7, 9,

11, 18 and 22). Following are presented some CSFs

that require further improvements:

 Since standardized SOW (Statement of Work)

is not provided, defining a service evaluation

structure is somewhat useless; and a 16.67%

satisfaction is observed (CSF 4). We may

infer that the enterprise has not provided the

necessary policies for defining and

categorizing its services.

 The enterprise does neither coordinate nor

standardize methods for integrating tasks to its

services (CSF 6, with a 10% satisfaction).

This may originate failures at planning levels

and influence the efficiency and use of the

project’s resources.

 No integrated knowledge basis has been

defined for storing management and

performance reports from different suppliers

(CSF 22, with 25 % satisfaction). This may

lead to failures at knowledge management and

to misuse of experience that could be reverted

into other projects.

The pilot project shows failures for 5 CSFs (8,

10, 12, 16 and 21), which reached a 0% satisfaction.

Comments are as follows:

 CSF 8: No mechanisms and/or policies have

been defined for change management and

change support.

 CSF 10: The client does not encourage

understanding and integration among

providers, No development standards are

followed and no system integration

perspective is adopted;

 CSF 12: There are no signed corporate

agreements.

 CSF 16: The lack of a penalty policy and

requirements’ efficient management

summarizing the interest of both, the client

and provider, puts the project management

and quality product delivery at risk.

 CSF 21: The provider has no ability to

negotiate new services or improvements once

the project is initiated, thus affecting the

contractual relationship and leaving aside

quality aspects that might have an impact on

the development process and product/service

acceptance.

5 DISCUSSION OF RESULTS

Upon the usage of ITO-Tracker at evaluating

SIGMSUAF project, we noted that the tool provides

effective support to SMEs in the follow-up and

control of their ITO projects, by assisting project

managers and leaders in the decision-making

processes aimed at consolidating their strengths and

overcoming their weaknesses based on the results

obtained from this tool, which shows clarity,

efficiency and accuracy.

Outsourcing project participants who used

ITO-Tracker listed the following tool strengths:

 It allows identifying areas to be addressed in

accordance with CSFs. When a CSF does not

reach a 75% satisfaction, certain measures

should be adopted to improve its performance.

 It allows regular automated project evaluation

through follow up, correction and/or

maintenance in order to analyze their behavior

and evolution.

 ITO-Tracker complies with non-functional

requirements.

 It displays the project strengths and

weaknesses and allocates respective

responsibilities to both, client and provider.

 It is based on the premise that project success

does not depend on one single party, and it

highlights the importance of a higher

commitment, communication and

management between SMEs and their clients.

 ITO-Tracker represents an interesting

alternative for benchmarking, and may be

used for ITO projects’ development and

analysis.

The results of the pilot project evaluation did not

surprise evaluators, since they acknowledged that

the provider did not make its best effort, and that its

weaknesses as a client are susceptible of being

improved. Furthermore, they suggested that

ITO-Tracker included the following functionalities

in a new version:

 Ability to weight each CSF and all related

questions upon determination of those CSFs

with the highest impact according to the

project type.

 Improvement and inclusion of new reports

aimed at enhancing the results analysis and

guidance, by making them more

understandable to users.

ICEIS 2008 - International Conference on Enterprise Information Systems

174

6 CONCLUSIONS AND FUTURE

WORKS

ITO-Tracker main characteristics include: user-

friendly, fault-tolerant, high performance,

documentation, and restricted access. Being a CSF-

based tool that uses a GQM approach, some of its

components might be reused for designing other

tools under this approach and needing to automate a

CSF-based model.

SMEs taking part in this pilot project stated that

the use of ITO-Tracker in these organizations is

highly feasible, since it provides an alternative for

project management and diagnosis.

ITO-Tracker is also expected to provide support

to researchers, specifically to LISI Lab at USB,

which will be able to offer this tool to SMEs, while

maintaining an updated database on the status of the

ITO projects developed by these organizations. The

tool allows LISI Lab to monitor their evolution,

generate evaluation reports, study project

success/failure statistics and suggest effective

solutions for improvement of the tool and/or CSFs

model supported. This way, SMEs shall be able to

identify those aspects with superior performance,

thus representing an added-value when it comes to

determine which strengths sustain their

consolidation and future development.

ITO-Tracker has been used for the evaluation of

software development ITO projects. We recommend

to validate its efficiency in all four ITO project types

identified: data center, network, software

development and hardware support.

ACKNOWLEDGEMENTS

This research was supported by National Fund of

Science, Technology and Innovation, Venezuela,

under contract S1-2005000165.

REFERENCES

Austin, D., 2002. Understanding Critical Success Factor

Analysis. W. W. Grainger, Inc. W3C/WSAWG

Spring. Retrieved February, 2007, from:

www.w3.org/2002/ ws/arch/2/04/ UCSFA.ppt.

Basili, V., Caldiera, G. and Rombach, H., 1994. Goal

Question Metric Paradigm. Encyclopedia of Software

Engineering (Vol 2), 528-532. John Wiley & Sons.

Fairchild, A., 2004. Information Technology Outsourcing

(ITO) Governance: An Examination of the

Outsourcing Management Maturity Model. Hawaii,

EUA: Tilburg University. Retrieved September, 2006,

from: http://csdl2. computer.org/comp/proceedings/

hicss/2004/2056/08/205680232c.pdf.

Kruchten, P., 2003. The Rational Unified Process: An

Introduction. Third Edition. Addison-Wesley

Professional.

Lee, J., Huynh, M., Chi-Wai, R. and Pi, S., 2003. The

Evolution of Outsourcing Research: What is the Next

Issue. 33rd Hawaii International Conference on

System Sciences.

Mayer & Bunge, 2004. Informática LTDA. “Panorama de

la Industria Latinoamericana de Software”. Page 97.

Brazil.

Méndez, E., Pérez, M., Mendoza, L., Ortega, M., Nuñez,

C. and Fernández, M., 2007a. “Factores Críticos de

Éxito Tecnológicos para Outsourcing de Tecnologías

de Información: Modelo Conceptual”. Technological

Critical Success Factors for Information Technology

Outsourcing: Conceptual Model. LVII Annual

Convention of the Venezuelan Association for the

Advancement of Science - AsoVac. San Cristóbal,

Venezuela. November.

Méndez, E., Pérez, M. and Mendoza, L., 2007b. A Tool

for Evaluating IT Outsourcing Projects based on

Critical Success Factors. Venezuelan Meeting on

Information Technology and Software Engineering,

EVETIS'07, Porlamar, Venezuela. Financed by

FONACIT through project G-2005000165.

O'Brien, J. A., 2005. Introduction to Information Systems.

Nueva York, NY: McGraw-Hill.

Rivas, L., Pérez, M., Mendoza, L. and Grimán, A., 2007.

Software Development Tools: Towards the

Construction of an Ontology. Venezuelan Meeting on

Information Technology and Software Engineering,

EVETIS'07, Porlamar, Venezuela. Financed by

FONACIT through project G-2005000165.

Rivero, D., Montilva, J., Granados, G., Barrios, J.,

Besembel, I. and Sandia, B., 2007. The Software

Industry in Venezuela: A Characterization of Human

Resources. Venezuelan Meeting on Information

Technology and Software Engineering, EVETIS'07,

Porlamar, Venezuela. Financed by FONACIT through

project G-2005000165.

Sheehy, D., Baker, G., Chan, D., Cheung, A., Grunberg,

H. and Henrickson, R., 2003. Information Technology

Outsourcing. Toronto, Canada: The Canadian Institute

of Chartered Accountants. Retrieved September, 2006,

from: http://www.cica.ca/multimedia/Download_Libra

ry/Research_Guidance/IT_Advisory_Committee/Engli

sh/eIToutsou rcing0204.pdf

Temnenco, V., 2006. UML, RUP, and the Zachman

Framework: Better together. IBM. Retrieved June,

2007, from: http://www.ibm.com/developerworks/

rational/library/ nov06/temnenco/#author

Veit, M. and Herrmann, S., 2003. Model-View-Controller

and Object Teams: A Perfect Match of Paradigms. 2nd

International Conference on Aspect-Oriented Software

Development, Boston, March.

ITO-TRACKER - A Tool for Evaluating ITO Projects based on Critical Success Factors

175