THE IMPACT OF COMPETENCES ASSESSMENT SYSTEMS

TO FIRM PERFORMANCE

Study on Project Management e-Assessment

Maria-Iuliana Dascalu

, Camelia Delcea

, Dragos Palaghita

and Bogdan Vintila

Department of Economic Informatics, University of Economics, Calea Dorobanti, no.15-16, Bucharest, Romania

Department of Cybernetics, University of Economics, Calea Dorobanti, no.15-16,Bucharest, Romania

Keywords: Assessment Systems, Project Management, e-Learning, Firms’ Financial Performance, Firms’ Non-financial

Performance

Abstract: Current paper studies the relationship between the use of e-assessment in the field of project management

and firms’ performance. For this purpose, a case study was especially designed to explore whether the

extensive use of an e-assessment application influences the financial and non-financial indicators of firms’

performance. The participants in the study were the users of the e-assessment application, employees

working in Romanian firms, from IT, education and consulting business sector. The main findings of the

study were the positive correlation between the considered performance indicators and companies’ growth.

The study is presented in the framework of learning at work and highlights the importance of using

information systems in enterprise environments, to develop professional competences and, thus, to achieve

business excellence.

1 INTRODUCTION

Firm performance can be defined as the difference

between the value that managers expect to obtain

from the productive activities and the value that a

firm truly creates (Chen et al., 2008). There are

various explanations for firm performance in the

literature. The resource-based approach explains the

firm performance as coming from inside the firm,

not given by market power (Harris and Ogbonna,

2001). The strategy-based approach highlights the

role of learning and rapidly changing as an

advantage over competitors and, thus, a factor for

obtaining performance. Both theories stress the

importance of human competences in achieving

business performance. A recent study reveals that

many “firms have embraced the notion of human

capital as a good competitive advantage that will

enhance higher performance” (Marimuthu et al.,

2009, p. 1).

The relationship between information systems

and the development of competences and, implicitly,

the development of human capital, was explored by

many researchers so far (Zhang and Lado, 2001),

(Delcea and Dascalu, 2009). In a previous study,

some tools for reaching competitive advantage by

implementing knowledge strategies were identified

(Delcea and Dascalu, 2009). Among them, a special

place was taken by e-learning and e-assessment

tools.

Current paper analyzes o sample of small and

medium Romanian firms which extensivelly used an

e-assessment tool for project management

competences. How the e-assessment tool affected

the performance of the considered firms is the

research question of our case study. Correlations

between the features of the e-assessment application

and their effects on business performance are also

underlined.

2 REASONS FOR LEARNING IN

FIRMS

As a consequence of quick changes, firms ask for

employees to be more involved in learning activities:

“employees need to take greater personal

responsibility to ensure their skills are current and

marketable” (Garofano and Salas, 2005, p. 282). In

business contexts, learning is meant to multiply the

Dascalu M., Delcea C., Palaghita D. and Vintila B. (2010).

THE IMPACT OF COMPETENCES ASSESSMENT SYSTEMS TO FIRM PERFORMANCE - Study on Project Management e-Assessment.

In Proceedings of the 12th International Conference on Enterprise Information Systems - Software Agents and Internet Computing, pages 59-64

DOI: 10.5220/0002864200590064

 SciTePress

activities of economic value. For learning to conduct

to professional competences, it’s better to be

integrated in an organizational environment or in a

community of practice (Lundin, 2005). Advices

from co-workers, as well selection for suitable

activities for competences development has a

tremendous impact on firm performance (Kröll,

2007).

Firm performance can be quantified by:

• Financial Indicators: productivity, market share

or profitability;

• Non-financial Indicators: customer satisfaction,

innovation degree or skills development maturity

(Marimuthu et al., 2009);

Learning activities influences more than one

indicator. For example, profitability, which is a

financial metric, is determined by the strategic

capabilities of a firm. The accumulation of

capabilities is correlated with the rate of learning

(Leiponen, 2006). On the other hand, one of the

measures for learning quality is its degree of

transformation: the increase of competences as a

result of the learning process. This aspect impacts

on the maturity of skills development in firms,

which is a non-financial indicator of performance.

3 ROLE OF ASSESSMENT

SYSTEMS IN LEARNING

ACTIVITIES HELD BY FIRMS

In enterprise environments, assessment has cubic

valence (see Figure 1): it is used both in decision

processes – recruiting, periodic evaluaiton and in

learning activities, which are more and more

diversified (Trausan-Matu Marimuthu et al., 2009).

Because e-learning platforms are expensive or

training activities consume a lot of time, many

enterprises prefer to use e-assessment services for

both purposes: checking employees’ knowledge and

making them realize what they don’t know and what

they should learn by themselves (e-assessment used

as a knowledge gap detector). E-assessment, as a

form of competence development, can be used to

increase firm performance (Leiponen, 2006), but

also as a tool to fill the symptomatic matrix in

diagnosing firms’ diseases (Delcea and Scarlat,

2009), as a strategic instrument.

Taking into account the cubic valence of e-

assessment for a firm, we studied whether these

general assumptions are valid also for project

management, a domain with a higher degree of

applicative parts.

Figure 1: e-Assessment Valences in Firms.

4 A CASE STUDY REGARDING

PROJECT MANAGEMENT

E-ASSESSMENT AND ITS

IMPACT TO FIRM

PERFORMANCE INDICATORS

As project management is the main form of

economic organization in knowledge society, there

is a high connection between firms’ performance

and projects’ success. Extensive research has been

done on this topic, but project management is a

highly dynamic field. This aspect is proved by

numerous standardized guides and codes of

practices, such as PMBOK (Project Management

Institute, 2004), developed by Project Management

Institute (PMI) or IPMA Competence Baseline

(ICB) (International Project Management

Association, 2006), developed by the International

Project Management Association (IPMA). These

codes of practices are continuously being improved.

The current study relies on the ICB approach,

which states that one can perform high project

management related activities if one holds technical,

behavioral and contextual competences. It is difficult

to choose the proper tools for assessing these kinds

of competences. The certification process organized

by IPMA relies on:

• Written Exam: several types of questions

(multiple choice questions, direct text

questions, open essays, intellectual tasks) with

ICEIS 2010 - 12th International Conference on Enterprise Information Systems

reference to the IPMA Competence Baseline

(International Project Management

Association, 2006), to be answered by the

individual in a limited period of time;

• Report: covering the subjects that describe the

management of a real project, programme or

portfolio (depending on the IPMA Level), with

reference to the ICB, on an appropriate number

of pages;

• Workshop: problem solving as part of a small

team on an example project, observed by one

or several assessors, revealing the actions of

the individuals in different roles, especially the

role of the (sub-) project manager;

• Interview: answers to specific, representative

questions prepared on the basis of the

candidate’s report (can be extended to the self-

assessment, written exam questions, workshop

results, replies from referees) with reference to

the ICB;

Due to the spreading of information systems, written

exams have been replaced with computer based tests

by many national bodies (members of IPMA). The

Romanian Association of Project Management

offers an e-assessment application as a preparation

tool for the real certification exams: “CertExam”

(http://pm.org.ro/certexam). Romanian Association

of Project Management is successfully affiliated to

IPMA, so the e-assessment tool respects the ICB

content.

The aim of our study was to check whether

“CertExam” is considered efficient in business

environments: we explored whether the knowledge

checked by the e-assessment tool used by the

Romanian Association of Project Management

influences the firms’ performance.

4.1 Research Methodology

The case study design was based on Yin’s book

(Yin, 2003). The research question, as previously

mentioned in the paper, was: “How e-assessment in

project management influences the performance of

Romanian firms?”. The objects of the study were a

few Romanian firms’ performance indicators and

“CertExam” e-assessment application (see Figure 2).

The participants were the users of “CertExam”, the

ones who took the tests. As instruments of research,

we mainly used the questionnaire and, sometimes,

interviews and informal e-mails.

The propositions of our study were strongly

related to the following performance indicators:

• Wage Increases;

• Customer Loyalty Rate;

• Number of employees who obtained professional

certifications;

The performance indicators chosen by us are also

reflected in other theoretical models (Marimuthu et

al., 2009): wage increases is a reflection of

profitability, number of returning clients is a form of

calculating customers’ satisfaction and number of

certified employees is an expression of skill

development maturity.

The propositions of our study are stated below:

• There is a connection between the use of

“CertExam” and the wage increases for the users

of the e-assessment application, the number of

clients who returned to the firms and the increase

in the number of certified personnel.

• The impact of the e-assessment application on

firm’s performance depends on the business

sector of the firm.

4.2 Participants’ Profiles

In order to establish the relationship between the use

of “CertExam” (the dependent variable) and

Romanian firms’ performance (the independent

variable), we questioned the users of the application.

The users can be classified according to the level of

their abilities in project management activities: from

level A-D (International Project Management

Association, 2006).

Questionnaires were sent to 153 users of the

application. The response rate was 54.25%.,

meaning that 83 users responded back.

THE IMPACT OF COMPETENCES ASSESSMENT SYSTEMS TO FIRM PERFORMANCE - Study on Project

Management e-Assessment

Figure 2: An e-Assessment Application for Project Management Competences.

When establishing the respondents’ profiles, we took

two aspects into consideration:

• The level of e-tests accessed by the users;

• The affiliation of the users to the same firm or

the same business sector;

Details about the respondents’ profiles can be

consulted in Table 1. The main method of analysis

was the pattern matching between the current

statements in the literature (Marimuthu et al., 2009)

and our findings from the empirical study. We

obtained some data triangulation from the

participants who stated that they worked in the same

firm.

Table 1: Profiles of "CertExam" Users.

E-Tests Level

IPMA Level No. Users No. Respondents

A 0 0

B 11 2

C 35 12

D 106 69

Affiliation to Business Sector

Business sector No. Firms No. Participants

IT 7 32

Consulting 9 29

Education 3 12

Other sectors 2 10

4.3 Main Results

Our study revealed that 41% of our respondents got

an IPMA certification as a result of their use of

“CertExam”, most of them from the IT sector. The

reason for starting using “CertExam” was their

desire of getting certifications. This desire drifted

from the need of getting professional recognition, a

higher level of well being and improving the firms’

portfolios. The fact that most individuals who

finalized their work with the e-assessment tool were

from the IT sector proves the striving force which

pushes the Romanian IT firms to achieve greater

performance.

Just 9% of our participants got a wage raise after

using the e-assessment tool. This low result can be

caused by the recession. 50% from those who got

the raise named the competence development

through the e-assessment tool as one important

factor for it. The ones who made this affirmation

were from different business sectors.

Most of our participants said that the customer

loyalty rate in a six-month period was of 61%,

which is a very high percent. Most respondents who

stated this fact were from the consulting sector.

When we asked our participants if they thought that

their improved skills, as a result of the “CertExam”,

had any impact on clients’ loyalty rate, 62% of them

said a definite “yes” and the other 38% agreed on a

ICEIS 2010 - 12th International Conference on Enterprise Information Systems

Figure 3: Knowledge Creation Function in e-Assessment.

certain level of impact. The users from education

sector (training organizations) named the

certification gaining as the main stimulus for their

clients to call again their services.

4.4 Critical Analysis

The results of our case study proved that might be a

link between the use of “CertExam” and the growth

of performance indicators at firms. We claim that

this relationship is not pure speculation, by

specifying that other researchers underlined the

importance of continuous competences development

to firm growth (Hitt et al., 2000), (Kröll, 2007),

(Zhang and Lado, 2001).

It is difficult to make a comparison between the

analyzed e-assessment tool and other tools, as “the

degree to which the advantages and disadvantages of

e-assessment are manifest is dependent upon how e-

assessment is deployed and embedded

organizationally” (

Kröll, 2009, p. 549). Nevertheless,

e-assessment was identified, by our respondents, as

less efficient than e-learning, from the learning

perspective, but more useful in enterprise

environments, from the strategic perspective.

5 FUTURE WORKS

Except the answer at our research question, we also

drew some other remarks which could be useful in

developing efficient competences assessment

systems. The participants of the study stated that

they found quite useful the feed-back mechanism of

the e-assessment application. This was designed to

create knowledge. As one could see in Figure 2, a

wrong answer is displayed in red and points out to

what should have been the right answers.

The e-assessment tool is not just an evaluation

tool, but a learning instrument, a knowledge creator,

as seen in Figure 3.

To increase the knowledge creator dimension, the

following features will be added to the e-assessment

tool:

• Adaptive behaviour;

• Increase in competences orientation;

An adaptive behaviour would provide not only a

better profile of the examinee, but also a better

assessment-taking experience and a much more

chance to fill the knowledge gaps which were

revealed by the e-assessment (Bodea and Dascalu,

2009). For matching the competences of the

employees to the ones considered as units in

electronic assessment, hierarchical structures, such

as semantic nets, will be used.

6 CONCLUSIONS

The paper provides an empirical study on the

relationship between the use of an e-assessment

application and the firms’ performance. Although

the study addresses only three indicators of business

performance, the positive correlation between the

assessment system and companies’ growth is

obvious. The fact that project management domain,

a key factor in economic development, is the

umbrella for the current research strengthens its

relevance. Further investigations on correlating e-

assessment use with other performance indicators

are the normal follow-up for the study. The current

work highlights the importance of using information

systems in enterprise environments (Raymond et al.,

2009) and also stresses the importance of developing

professional competences to achieve business

excellence.

THE IMPACT OF COMPETENCES ASSESSMENT SYSTEMS TO FIRM PERFORMANCE - Study on Project

Management e-Assessment

ACKNOWLEDGEMENTS

This article is a result of the project „Doctoral

Program and PhD Students in the education research

and innovation triangle”. This project is co funded

by European Social Fund through The Sectorial

Operational Program for Human Resources

Development 2007-2013, coordinated by The

Bucharest Academy of Economic Studies.

REFERENCES

Bodea, C. N., & Dascalu, M. (2009). Designing Project

Management Tests based on Semantic Nets and

Concept Space Graph. Creating Global Economies

through Innovation and Knowledge Management:

Theory & Practice, Proceedings of the 12th

International Business Information Management

Association Conference (pp. 1232 – 1236 ). Kuala

Lumpur: IBIMA.

Chen, L.-J., Chen, C.-C., & Lee, W.-R. (2008, 1).

Strategic Capabilities, Innovation Intensity, and.

Journal of Service Science and Management , 111-

122.

Delcea, C., & Dascalu, M. (2009). Knowledge Strategies

Tools for Managing Enterprise Crisis. The 4th

International Conference on Knowledge Management:

Projects, Systems and Technologies (pp. 115-117).

Bucharest: “Carol I” National Defense

University,Romania.

Delcea, C., & Scarlat, E. (2009). The Diagnosis of Firm's

Diseases Using the Grey Systems Theory Methods.

IEEE International Conference on Grey Systems and

Intelligent Services (pp. 755-762). Nanjing : IEEE.

Garofano, C., & Salas, E. (2005). What influences

continuous employee development decisions? Human

Resource Management Review, 15, 281–304.

Harris, L. C., & Ogbonna, E. (2001). Competitive

advantage in the UK food retailing sector:past, present

and future. Journal of Retailing and Consumer

Services , 8, 157-173.

Hitt, M. A., Ireland, R. D., & Lee, H.-u. (2000).

Technological learning, knowledge management,firm

growth and performance: an introductory essay.

Journal of Engineering and Technology Management ,

231-246.

International Project Management Association. (2006).

IPMA Competence Baseline. Nijkerk, Netherlands:

IPMA.

Kröll, M. (2007). Limits and possibilities for competence

development in firms. European Conference of

Educational Research (ECER). Ghent.

Kröll, M. (2009). E-Assessment: A Suitable Alternative

for Measuring Competences?. Lecture Notes in

Computer Science, 5613, 543-550.

Leiponen, A. (2006). Dynamic Competences and Firm

Performance. Laxenburg: International Institute for

Applied Systems Analysis.

Ling, F. Y., Low, S. P., Wang, S. Q., & Lim, H. H. (2009).

Key project management practices affecting

Singaporean firms’ project performance in China.

International Journal of Project Management , 59–71.

Lundin, J. (2005). Talking about Work: Desiging

Information Technology for Learning in Interaction.

Gothenburg: Gothenburg Studies in Informatics.

Marimuthu, M., Arokiasamy, L., & Ismail, M. (2009).

Human Capital Development and Its Impact on Firm

Performance: Evidence from Developmental

Economics. The Journal of International Social

Research , 2(8), 265-272.

Project Management Institute. (2004). A guide to the

project management body of knowledge (PMBOK

guide). Newtown Square, Pennsylvania: PMI Inc.

Raymond, L., Croteau, A.-M., & Bergeron, F. (2009). The

Integrative Role of IT in Product and Process

Innovation: Growth and Productivity Outcomes for

Manufacturing.

Enterprise Information Systems, 11th

International Conference (pp. 27-39). Berlin

Heidelberg: Springer-Verlag.

Trausan-Matu, S., Posea, V., Rebedea, T., & Chiru, C.

(2009). Using the Social Web to Supplement Classical

Learning. Advances in Web Based Learning – ICWL

2009 (pp. 386-390). Berlin Heidelberg: Springer-

Verlag.

Yin, R. K. (2003). Case Study Research: Design and

Methods (3

ed.). Thousand Oak, California: SAGE

Publications.

Zhang, M. J., & Lado, A. A. (2001). Information systems

and competitive advantage: a competency-base view.

Technovation , 21, 147–156.

ICEIS 2010 - 12th International Conference on Enterprise Information Systems