A Procedural Approach for Evaluating the Performance of Business

Processes

Thiago Mendes and Simone Santos

Centro de Informática, Federal University of Pernambuco, Av. Jorn. Aníbal Fernandes, Recife, Brazil

Keywords: Strategic Alignment, Business Process Management, Measurement Performance.

Abstract: BPM is a systematic approach that enables an organization to achieve results that are consistent and aligned

with its strategic objectives. In this context, organizations need to measure the performance of their processes,

thereby enabling them to support planning, inducing control and making it possible to diagnose the current

situation. The research that led to this article was prompted by the paucity of empirical investigations into

performance measures offered in the literature and recognition of the difficulties that organizations face when

attempting to verify the results of their business processes. From a systematic review of the literature, it

became evident that, among the approaches discussed, there are several variations in the methodology, in the

specifications of the measures and even as to how business processes are evaluated. By undertaking a

comparative evaluation of the approaches, a set of criteria was defined which addresses not only theory, based

on the references, but also some usability aspects of a process performance model. Thus, we proposed the

definition of a new approach that brings together elements and recommendations selected from the approaches

analyzed, plus enhancements. To evaluate the proposed approach, a case study is discussed, and important

results presented which demonstrate its applicability.

1 INTRODUCTION

Various organizations have been engaging on

reorganizing their business processes and

implementing process-based management in order to

gain competitive advantage. From this perspective,

the concept of Business Process Management (BPM)

has emerged as a management approach that shifts the

focus from functional units to controlling the

performance of business processes in achieving their

objectives.

BPM enables corporate processes to be

standardized, thereby increasing the productivity and

efficiency of the organization, Its aim is both to

monitor, after having defined the priorities of the

processes, how the resources of an organization are

applied and transformed into actions to achieve the

organizational goals (Baldam et al., 2007). The

results of the processes are directly linked to the

mission and goals of the organization, since the

processes represent the implementation of the

strategy.

The focus of process management is supported on

key business strategies that establish the direction of

the organization. Smith and Furt (2009) emphasize

that a critical factor is the need for measurement in

process management. Therefore, it becomes essential

to use measures that enable the performance of

processes to be monitored, thus contributing to check

how well these processes meet the strategic objectives

that have been set.

Trkman (2010) states that issues related to

performance measures and to defining what should be

measured in relation to business processes should be

directly linked to the strategic priorities of each

organization.

BPM initiatives need to be evaluated to check the

alignment between the strategic, tactical and

operational aspects of processes, thus making it

possible to verify the results achieved in accordance

with the objectives outlined. Business processes

should be designed by the administration, after

having established measures of performance (Powell

et al., 2001), which should reflect the desired

direction in strategic objectives, and serve as a basis

for the control of processes.

According to Kueng and Krahn (1999), as well as

Vuksic et al. (2008), the effects of BPM programs are

often not easily visible as to the generation of value

when organizations are unaware of or do not have

good control over the operation of its processes. The

518

Mendes, T. and Santos, S.

A Procedural Approach for Evaluating the Performance of Business Processes.

In Proceedings of the 18th International Conference on Enterprise Information Systems (ICEIS 2016) - Volume 2, pages 518-527

ISBN: 978-989-758-187-8

result of this is to create performance indicators that

reflect departments’ prompt results, defined by a

functional management focused on a vertical view

(Pidun and Felden, 2012). Thus, measures and

evaluation of performance end up targeting the

functional performance of departments and

individuals when they should focus on the outcomes

of the process.

According to Leclair et al. (2012), organizations

have difficulty in defining what should be measured,

and end up measuring less complicated aspects such

as productivity, cost, time; and often neglecting

indicators linked to strategy. Based on their analysis

of empirical studies, Pidun and Felden (2012) verified

there is a gap between the analysis, implementation

and execution of processes as well as between

strategic management and operational execution; to

which they give as a possible explanation the lack of

a methodological orientation of systems that measure

the performance of Business Processes.

In the analysis of Pidun et al., (2011), various

models for assessing the performance of Business

Processes use numerical parameters, and artificial and

simplifying measures in an attempt to assess

processes. However, various processes of a

qualitative and non-deterministic nature end up not

being evaluated effectively. If various processes are

not evaluated or this evaluation is difficult when

using these measures, possible problems or

performance results may remain invisible to

managers, and thus make hardly any contribution to

the decision-making process and consolidate the gap

between strategy and business processes (Pidun and

Felden, 2012). This leads us to following research

question: Which approach best meets the evaluation

of business processes, with a view to a greater

alignment between process indicators and strategic

objectives?

Based on the model proposed by Wohlim (2000),

which was adapted from the GQM (Goal Question

Metric) method, the overall objective of this research

is: (1) To analyze metric models or approaches for

assessing business processes proposed in the

literature; (2) Proposing an assessment approach to

business processes that contribute to a better

alignment between indicators of processes and

strategic objectives; (3) Checking applicability from

the standpoint of leaders, analysts and owners of the

process; (4) In the Business process of a public

organization context.

The rest of the article discusses theoretical and

methodological reference works in Section 2. Then,

the Approach put forward in this paper is described

(Section 3). Section 4 presents the results obtained by

applying the Approach and in Section 5 final remarks

are made, the contributions of the paper listed and

suggestions for future studies made.

2 THEORETICAL AND

METHODOLOGICAL

REFERENCES

The research began with an ad hoc review of the

literature in order to search for and acquire, in

general, a better understanding of how the

performance of business processes is evaluated. This

review contributed to defining the research problem

and subsequently to formulating the objectives of this

paper.

After this stage, it was found there was a need to

conduct a systematic review of the literature (SRL).

This method enables a wider range of relevant results

to be included, instead of the limitation on our

conclusions that would result from reading only a few

articles. The bibliographical search was conducted in

accordance with the systematic review of literature

method defined by Kitchenham (2004).

With the objective of analyzing, more specifically,

what the measures or indicators of they are and how

they are being used measures or indicators of

performance are and how they are being used in the

context of managing business processes to ensure that

processes and strategic objectives are aligned. The

following search questions for the SRL were defined:

 QP1- What are the metrics and indicators that

are being used in the evaluation of business

processes?

 QP2- Do the studies have rules, guidelines or

sets of guidance on how to use the metrics

presented?

 QP3- What is the context in which the metrics

are being used?

Automatic searches were conducted in the digital

libraries of the Association for Computing Machinery

(ACM), the IEEE Computer Society, Emerald

Insight, Science Direct and Springer Link. The result

of the searches resulted in the return of 3,377 articles.

Their titles and/or abstracts of these were analyzed,

and exclusion criteria were applied. References were

discarded that clearly dealt with other matters not

relevant to the purpose of this research, as a result of

which an initial selection of 25 items was made. The

second filter consisted of reading the text of the

articles initially selected in their entirety and as a

result the list of articles described in Table 1 were

obtained.

A Procedural Approach for Evaluating the Performance of Business Processes

519

It was found from the analysis of the approaches

selected (Section 3), there was a need to propose a

systematic approach (Section 4), which would

combine relevant and complementary aspects of the

articles analyzed.

Table 1: Final result of selecting the studies.

ID Title of Article

EP6 Optimizing Process Performance Visibility

through Additional Descriptive Features in

Performance Measurement.

EP7 Organizational Performance Measures for

Business Process Management: A

Performance Measurement Guideline.

EP9 Research on Key Performance Indicator

(KPI) of Business Process.

EP11 The Research of Metrics Repository for

Business Process Metrics.

EP13 Two Cases on How to Improve the

Visibility of Business Process Performance.

EP15 Performance measurement in business

process outsourcing decisions: Insights

from four case studies.

EP22 Quality evaluation framework (QEF):

Modelling and evaluating quality of

business processes.

In order to verify the usefulness of the proposed

approach, a case study was conducted of a business

process of a public organization. In particular, this is

an empirical research that according to Yin (2005)

that investigates a current phenomenon within its real

context, when the boundaries between the

phenomenon and context are not clearly defined. The

case study is described in Section 5 of this article.

The different approaches discussed above show

variations in the methodology, specifications and

even as to how business processes are evaluated. In

order to assess these approaches, a set of criteria was

defined which addresses both the theory, based on the

selected references, and some aspects of usability.

The criteria used were:

 Methodology: Ad-hoc (Ah) OR Systematic

(Sy);

 Types of measures: Quantitative (Qt) AND/OR

Qualitative (Ql);

 Context of Applications: Specific (S) OR

Generic (G);

 Processes supported by systems: Yes OR No;

 Efficiency: Yes OR No;

 Effectiveness: Yes OR No;

 Empirical validation: Yes OR No.

The criterion of Methodology is related to the

attention to the way in which an approach can be used

to evaluate a business process. Classified approaches

such as systematic ones are those that describe a set

of rules, guidelines, processes or activities needed to

use the measures presented. On the other hand, ad-

hoc approaches focus only on describing performance

measures, without, at first being interested in the way

that its approaches can be put into practice.

The criterion named types of measures concerns

the nature of the measures presented in the selected

approaches. Quantitative measurements are those

based on numerical performance indicators, while

qualitative measures consist of textual descriptions

and narratives about factors of success of the process

and which often require interpretation.

The criterion called application is related to the

context in which a particular approach can be used.

Certain approaches present measures that are

sufficiently generic so that they that can be applied in

different contexts and business processes of very

different natures. Approaches classified as specific

are those used in a specific business process or those

of a similar nature.

Some approaches have performance measures

that are established from information generated by

business process automation systems. In other words,

the use of the measures presented is associated with

making information available by means of systems.

The criterion of processes supported by systems is

related to the concern for analyzing if the

performance evaluation of the process depends, in

principle, on some support by means of systems such

as a BPMS (Business Process Management Suite), for

example.

The aim of using criteria of efficiency and

effectiveness is to describe whether the approaches

analyzed present measures to evaluate the

productivity and performance (efficiency) of

processes, as well as their ability to do what is needed,

which is correct in order to reach a certain goal or

outcome (effectiveness). Efficiency involves the way

in which an activity or process is performed;

effectiveness refers to whether it results in meeting

customer’s needs in all their restrictions. According

to Corrêa and Corrêa (2006), efficiency is a measure

of the extent to which an organization's resources are

used economically, and effectiveness refers to the

extent to which the objectives are achieved.

The criterion of empirical validation is concerned

with assessing the practical utility of the proposed

measures in the different approaches. Empirical

validation occurs by conducting experiments, case

studies or research in a real context, and helps to

determine the effectiveness of these measures.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

520

Tables 2 and 3 present the approaches

investigated related to the evaluation criteria

illustrated above

Table 2: Evaluation Criteria (Methodology, Measures and

Application).

Articles

Evaluation Criteria

Methodology Measures Application

Ah Sy Qt Ql S G

EP6

X X X X

EP7

X X X

EP9

X X X

EP11

X X X

EP13

X X X X

EP15

X X X

EP22

X X X

Table 3: Evaluation Criteria (System support, Efficiency,

Effectiveness and Empirical validation).

Studies

Systems Efficiency Effectiveness Validated

EP6

No Yes Yes

EP7

No Yes Yes

EP9

Yes Yes Yes

EP11

Yes Yes No

EP13

No Yes Yes Yes

EP15

No Yes Yes Yes

EP22

No Yes No Yes

Based on the criteria presented and discussed

earlier in this paper, we considered that a systematic

approach (which enables the metrics presented to be

used consistently), which brings together quantitative

and qualitative performance measures, generically

(measures not linked to a specific context or domain)

not dependent on using systems that provide an

evaluation of efficiency (resources) and effectiveness

(results) and which had been validated empirically

could be considered ideal, or one that best meets the

evaluation of performance of business processes.

From the analysis of the selected approaches,

there was a need to propose a systematic approach

that would combine relevant and complementary

aspects of the studies analyzed.

3 APPROACH PROPOSED FOR

EVALUATING THE

PERFORMANCE OF BUSINESS

PROCESSES

Of the approaches investigated, the one that

comprises the largest number of requirements, set out

in the previous section, is Article EP13. However, it

does not describe in great detail how this approach

can be used. Moreover, the approaches presented in

articles EP7 and EP22 do give a detailed description

of the the process for using performance measures.

In article EP7, Vuksic et al. (2008) argue that the

performance measures of a process should be aligned

to the strategic goals and objectives of the

organization and that they should be measured under

the quantitative and qualitative dimensions,

Nevertheless, the metrics presented in the article are

quantitative, whereas, in Article EP22, the

performance measures are associated exclusively

with expressions quantified as a quality objective.

It was arising from these considerations that our

approach was defined. It brings together the model of

performance measures described in EP13, but using it

as a guide to using the measures presented, and the

procedural description defined in Article EP7.

3.1 Process for Evaluating the

Performance of Processes

The evaluation process defined in EP7 is illustrated in

Figure 1 below:

Figure 1: Process for using performance measures.

The aim of the activities illustrated in the process

flow is to lead to proposing a guideline for measuring

the performance of business processes. In order to

provide adequate guidance for assessing

performance, on following this procedural

description, a description will be given below of the

steps and some artifacts or suggestions that may

contribute to making such an assessment.

3.1.1 Defining the Strategic Objectives

According to Vuksic et al. (2008), the performance

measures of business processes should be aligned to

the organizational goals and objectives. The first

activity in the process of evaluating the performance

of a process consists of defining the strategic

objectives to be achieved by the results of the

processes. These objectives are defined by the owners

A Procedural Approach for Evaluating the Performance of Business Processes

521

of the process in the organization, and can be obtained

from the strategic planning documents.

3.1.2 Identifying the Stakeholders

After the identifying the objectives of the process, the

stakeholders in the conduct of the process must be

identified, i.e., those responsible for the process,

those taking part in the activities, clients,

beneficiaries, sponsors and others. Process mapping

artifacts or even the use of a RACI matrix can

contribute to defining the stakeholders in

implementation.

3.1.3 Defining the Information Required for

Each Party

The information needs of each stakeholder can be

very different, and require that the performance

measures be adapted to their requirements.

The activity of mapping makes it possible to

define the roles and responsibilities of each

participant during the implementation of the process

more clearly and objectively thereby contributing to

defining the information required for each party and

assisting in defining the performance measures.

3.1.4 Establishing what Will be Measured

In this stage what is sought is to identify what

measures are useful for evaluating the performance of

the process in relation to its goals, in achieving the

strategic objectives. Performance reports or

accounting documents can be used to identify

performance measures of the process.

3.1.5 Specifying how the Measures Will be

Collected

When identifying the performance measures of a

process, a process for collecting these measures needs

to be identified. In the case of processes supported by

systems, there automated data collection. In order to

develop an action plan for identifying and collecting

performance measures in non-automated processes,

an adaptation of the 5W2H management tool was

proposed as an artifact for this step. The information

is organized as follows:

 Description of the measure (What): Defines

what this measure means.

 Rationale (Why): Defines the end, the purpose

of using the measure.

 Instruments to obtain measures (From where):

Specifies the instruments (media) from where

the data of the measures can be collected.

 Frequency (When): Determines the frequency

or time interval at which the information of the

measures needs to be collected.

 Responsible (Who): Describes the sector or

department responsible for providing

information on a particular measure.

 Process for obtaining measures (How): Details

what process will be undertaken to obtain the

measures.

 Costs (How much): Describes the costs for

obtaining the measure.

3.1.6 Defining what Approaches, Methods

or Instruments Will be used

There are several approaches and frameworks that use

different methods for measuring performance,

described in the literature, as well as various tools that

support these evaluation methods.

Based on the critical analysis of the approaches

presented in Section 3, it was found that the model in

EP13 has the largest number the criteria presented.

Thus, it is recommended that this is used in step for

evaluating the process. Subsection 4.2 presents how

the measures in the EP13 model are systematized.

3.1.7 Assessing what the Measures Present

The performance measures of business processes

should be related to (or the same) measures existing

in the organization that are used to monitor the

success of the strategy.

3.2 Adapting the Evaluation Model

The performance measures defined in Article EP13

require a greater capacity of interpretation due to the

way they are presented. The paper presents the

measurements in the form of questions such as: There

numerical parameters linked to quality, such as cycle

time, probability of failure or average of

interruptions. This could hamper their use by

individuals who evaluate their business processes.

Given this configuration, the measures model was

systematized in an attempt to make it orderly,

coherent, clear and understandable by the parties that

may use it. The model originally defined in EP13 was

discussed together with two BPM experts until the

model shown in Tables 4 and 5 was reached.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

522

Table 4: Systematization of the Measures Model

(Efficiency).

(*)

Efficiency

Indicators Types of Indicators

Key

Perfor-

mance

Indicator

(KPI)

- Time cycle of the process;

- Countable amounts of process

outputs;

- Number of returned processes;

- Number of processes carried

out;

- Downtime;

- Percentage of the budget used.

(**)

Indicators Types of Indicators

Process

Success

Factors

(PSF)

- Benefits, scope and steps of

the process;

- Information systems,

databases or interfaces used;

- Descrition of the success of

the process;

- Documents and information

produced and delivered.

(*) Numeric Parameters; (**) Descriptive Parameters.

Table 5: Systematization of the Measures Model

(Effectiveness).

(*)

Effectiveness

Indicators Types of Indicators

Metrics of

the

Process

(PMX)

- Probability of failure;

- Average number of

interruptions;

- Time-cycle approval meetings;

- Number of approval meetings;

- Number of approvers /

officers-in-charge.

(**)

Indicators Types of Indicators

Descrip-

tion of the

Compo-

nents

which

comprise

the

Process

(PO)

- Departments responsible,

informed or affected;

- Role of approvers or officers-

in-charge;

- Interfaces with other

departments;

- Conditions and restrictions;

- Initiators, beneficiaries and

customers of the process;

- Key people or information

system for a step of the process

or shared backup.

(*) Numeric Parameters; (**) Descriptive Parameters.

Also added to the model described above was an

auxiliary table, which describes in detail each of the

indicators presented. Such as, for example, “Process

Cycle time: corresponds to the time required for

performing the process, i.e. the time between the start

and end of the process”.

The proposed approach to evaluating business

process performance of this work consists of the

process for measuring performance, artifacts and

model of systematic measures described in this

section. In order to ensure the validation of the

approach, it was applied in a real business process of

an organization, and this is described in the next

section.

4 RESULTS AND DISCUSSION

The process that was selected to validate the proposed

approach is called Pro-equipment, linked to the Pro-

rectorate for research and postgraduate subjects

(PROPESQ) of a Brazilian Federal Institution of

Higher Education (IFES).

Pro-equipment procures equipment intended for

shared use in the structure of scientific and

technological research of post-graduate programs of

the recommended IFESs and is funded by the

Coordination Unit for Improving the Qualifications

and Experience of Higher Education Personnel

(CAPES), and is linked to the strategic objective of

stimulating the development of research. The

selection of this process, to validate the proposed

approach, occurred in a timely manner in view of the

demand for having it mapped and optimized,

requested through PROPESQ together with the IFES

Office for Processes to which the author of this

research is linked.

4.1 Defining the Objectives of the

Process

The first interview was attended by two

representatives of the Research Board (DPQ in

Portuguese) and three process analysts of the

Processes Unit of the institution were also present.

The two participants are directly responsible for the

performance of the process.

Initially a description of the following points was

requested:

P1 – “What is the Objective of Mapping and

Optimizing the Process?”

According to Respondent 2, the main problem of

this process is the lack of transparency of the

information on the results, when he states that:

E2 - "We have partial and not effective control,

for example, we know the input volume of resources,

we know how much CAPES makes available, and we

even know how much of the resources were

earmarked because the Accounts Department of

PROPESQ send us this information on a spreadsheet,

if we request it. From there on, we have no feedback,

we do not know if the teacher received it. "

A Procedural Approach for Evaluating the Performance of Business Processes

523

P2- “What is the Objective of the Pro-Equipment

Process?”

E1 - "To stimulate scientific production by

acquiring equipment intended for shared use in

postgraduate laboratories of the university."

4.2 Identifying Stakeholders in the

Process

P5 – “Who are the Stakeholders in the Process?”

Specifically those in charge (the owners of the

process), participants of the activities (sectors),

clients, beneficiaries and sponsors.

CAPES acts as the sponsor, the clients of this

process were defined as the teachers and/or research

groups. Finally, the participants in the activities when

the given process was being carried out were: the

Director of Research (DPQ/PROPESQ), the

Accounting Board (DC/PROPESQ), the Agreements

Sector (PROPLAN), the Legal Department (Office of

the Rector), the Rector, The National Purchases

Sector (PROGEST), the Importations Sector

(PROGEST) and the Publishing Sector (PROGEST).

4.3 Defining the Information Necessary

for Each Party

At this stage we were invited to the meetings, prior to

which the stakeholders described above, excluding the

sponsor of the process (CAPES). This was justified by

the fact that CAPES is configured as an entity external

to the University context in which one does not have

dominion over the rules and procedures used.

Four meetings were held with the following

representatives: 02 (DPQ/PROPESQ), 01 (DC/

PROPESQ), 01 (Agreements Sector/PROPLAN) and

01 (PROGEST). At this stage the process was modelled

collaboratively using the BizAgi Process Modeler.

The process starts from the release of the official

notice by CAPES. Then, the Board of Research

(DPQ) sets an internal schedule of activities before

submitting a single proposal.

Generally, the amount of resource requested by

the projects is greater than the amount initially made

available by CAPES. Therefore, the DPQ convenes a

committee to assess and recommend what projects

should be submitted. The end result of the projects

selected internally is announced and submitted to the

CAPES.

These projects will be further evaluated by

CAPES, who may refuse some requests. Projects

approved at this stage are announced by DPQ. After

this step, the term of decentralization of credit is sent

to PROPLAN, the process is reviewed, and the

document is sent to the Legal Department, which will

analyze items such as dates, terms, rubrics, data, etc.

The Legal Department must give its assent in writing

to ensure the process continues. If so, the process is

sent for the signature of the Rector of the institution.

The process for requesting ear-marking is then

returned to PROPLAN which monitors that CAPES

has released the funds and PROPLAN notifies the

Accounting Sector of PROPESQ.

The Accounting Sector of PROPESQ then starts

the activity of mounting the ear-marking process for

each piece of equipment. At this moment, a request is

made to the coordinators of the subprojects for a

series of documents. Subsequently, the accounting

entry for the committed funds is made via PROGEST,

and there follows a new phase of analysis by the Legal

Department. After being approved by the Legal

Department, the process goes to the Publishing Sector

in PROGEST, and the flow of the process proceeds to

the National Purchases or Importations Sector so that

the purchase can be made.

4.4 Defining what Will be Evaluated in

the Process

At this stage there were two interviews with those

responsible (DPQ) for this process. Initially the

interviewee was asked about the existence of an

accounting report on the performance of the process.

According to E1 - "CAPES demands an indication

of how the previous buying process was conducted.

We describe, when asked, only what was actually ear-

marked .".

For interviewee E2, this kind of report is based on

measures that do not reflect, in a satisfactory manner,

that achieving the objectives of the process was verified.

E2 - "Ear-marking is only the first stage of the

expenditure budget, where the funds were granted, but

there is no guarantee the equipment will be purchased."

When asked: “What are the important

performance measures for making a satisfactory

assessment of the results of this process?”

E1 - "In my opinion we should have information

about whether or not the equipment was purchased,

was actually installed in the laboratory and which

research groups are using it".

Still on the measures that should be used to

evaluate the process, interviewee E2 stated that:

E2 - "Besides the amount ear-marked and bits and

pieces to be paid, which are data that we can get easily,

we need to know the quantity and description of the

pieces of equipment bought, how many and which ones

are awaiting delivery and installation, and what the

institutional outcomes are (number of dissertations,

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

524

theses, descriptive reports on patents generated or

information on the rendering of research services to

other institutions) that have been achieved".

4.5 Specifying how the Measures Will

be Collected

At this stage a meeting was held with the same

representatives who took part in the mapping of the

process meetings. Initially, a projection of the

mapping process was presented, followed by a set of

measures that were extracted. The measures

presented to the participants were:

 Total amount ear-marked;

 Amount spent the ear-marked resources;

 Amount in smaller bills to be paid;

 List of equipment purchased;

 List of delivered equipment;

 List of installed equipment and;

 Academic indicators.

For each of the measures proposed the artifact for

specifying the collection process defined in the

approach was discussed jointly by the participants

and filled in. Table 6 shows an example.

Table 6: Collection process for academic indicators.

Academic Indicators

Description

measure

Corresponds to descriptive reports on

data from the scientific production

supported by the equipment purchased.

Reports may contain descriptive

information about numbers of

dissertations, theses, patents obtained

as a result of using the equipment, as

well as descriptions of services

rendered to other institutions.

Justification

One of the strategic objectives of

PROPESQ is to encourage scientific

production of the university. Products

purchased by Pro-equipment are

intended to contribute to achieve this

goal. Academic indicators enable the

results of this process to be made more

visible in relation to achieving the

strategic objectives.

Instruments

to obtain

measures

Scientific production reports from

research laboratories that have

equipment coming from Pro-equipment

funds.

Periodicity Ad hoc.

Officer-in-

charge

Coordinators of postgraduate

laboratories.

Process for

obtaining

measures

Meetings or requests via email.

Costs Time

4.6 Applying Performance Assessment

Model

At this stage, four meetings were held with the

members already described: (02) representatives of

DPQ /PROPESQ, (01) representative of the

Accounting Sector/PROPESQ, (01) representative of

the National Purchases Sector and the Importations

Sector of PROGEST. Meetings were held separately.

The procedure for conducting this step may be

described in three phases:

 Presentation of the Model – in this stage the

presentation was made, using a print document

format, of the model of measures defined to the

interviewee.

 Recording of the participants' responses – the

interviewees’ answers were recorded in the

document.

 Summary of the responses – in this stage, the

responses were combined in a single table,

thereby eliminating redundancies and

inconsistencies in a single frame.

Table 7 presents some of the indicators obtained in

this step:

Table 7: Systematization of the Measures Model

(Efficiency).

(*)

Efficiency

Indicators Types of Indicators

Key

Perfor-

mance

Indicator

(KPI)

- Amount raised from CAPES;

- Amount of funds spent; ear-

marked (Percentage used of the

budget)

(**)

Indicators Types of Indicators

Process

Success

Factors

(PSF)

- Steps: Official notice

published, term of

decentralization of credit

unundertaken, Review and

analysis of the process, Release

of Credit Authorized,

Constructing ear-marked funds,

review and analysis of the funds

ear-marked followed by legal

approval, Publication in the

Official Gazette of the Union,

Purchase and control of

acquisitions made;

- Information systems, databases

or interfaces used: SICAPES

(the CAPES Integrated System),

net purchases (Purchases Portal

of the Federal Government),

SICAF (System for the Unified

Registration of Suppliers);

(*) Numeric Parameters; (**) Descriptive Parameters.

A Procedural Approach for Evaluating the Performance of Business Processes

525

4.7 What the Measures Present

A set of measures for the process as a whole, and its

"partitions" (steps, departments, stages, etc.) have

been defined in the previous section so as to reflect

certain performance characteristics for each

management level of stakeholder. These measures

describe by means of generating information the

actual state of the configuration of the process, thus

enabling aspects of performance to be evaluated such

as s complexity in operations, bottlenecks, redundant

activities, excessive documentation and approvals. In

addition, they make it possible to evaluate the results

of the process as to achieving the objectives set.

Regarding Pro-equipment, a check was made on

the opportunity to insert some improvements into the

process. The following optimization proposals were

discussed, as a result:

 Integrating the Superintendence of Works (SPO

in Portuguese) in the early stages of the process,

specifically in the (internal) evaluation stage of

the sub-projects to be submitted to CAPES. The

function of the SPO is to carry out assessment in

order to identify if the laboratory to which the

equipment will go, has the infrastructure needed

for its installation. In several situations it was

reported that the purchase of equipment had

been made but the research lab did not have an

infrastructure appropriate for its installation.

Currently, the process for guaranteeing

resources for renovations, or purchase of

equipment to ensure the effective installation of

the equipment is only started from the moment

that the equipment is delivered to the university.

 The urgency to get the guarantee of these funds

for the purchase of equipment by ear-marking

funds encouraged the actors of the department

responsible not to check the (full) requirements

of documentation defined by legislation for the

formation of these processes (ear-markings).

The process very often did not follow its

"normal" flow, where and if it was published in

the Official Gazette of the Union without

necessarily obtaining the approval of the Legal

Department. This deviation in the flow led, in

various situations, to delays in conducting the

process due to non-observance of the applicable

legislation.

 The information on the results obtained for the

research based on acquiring equipment were

practically non-existent or difficult to monitor.

As a proposed solution which was discussed

between the actors of the process is the creation

of the role of research lab coordinator.

At first, the adoption of the model, which

combined quantitative performance measures (that

permit performance to be measured and managed)

and qualitative measures (which allow grounding of

the critical analysis of the results), manages to gather

important information about the performance of the

process in relation to the strategic goals and

objectives set, in accordance with the evaluation of

utility perceived by the participants in the process.

5 CONCLUSIONS

In relation to the overall objective of the research,

which consisted of verifying which approach best

serves the evaluation of business processes, with a

view to greater alignment between process indicators

and strategic objectives, it may be noted that there is

no single approach that adequately assesses the

performance of business processes.

Several variables can be measured and evaluated

with regard to business processes. However, it falls to

managers to undertake the tasks of identifying,

selecting and defining measures that are adequate for

and aligned with the organization's objectives. The

model used has a large number of indicators that can

and should be adapted to different organizational

contexts, from which the most important for use in

practice should be selected. Finally, it is essential that

an organization uses several indicators when

evaluating its business processes, since the use of a

single indicator can not represent the broader context

needed to support effective decision making.

Note that in the analysis of the literature and in the

very conduct of the empirical study, there was evidence

of the need to evaluate the performance of business

processes based on quantitative and qualitative

measures. Managers must consider the various existing

measures for evaluating business processes and should

spare no effort in identifying, selecting and defining the

most important one for use in practice.

In order to complement the results found in this

research, we propose the use of the approach in

several different business processes, with a view to

verifying its real results for the different

characteristics of operations and dynamics of an

organization such as: culture, size, area of activity,

and so one.

REFERENCES

Baldam, R. et al, 2007. Gerenciamento de processos de

negócios. 2ª edição. São Paulo: Editora Érica.

ICEIS 2016 - 18th International Conference on Enterprise Information Systems

526

Bosilj-vuksic, V.; Milanovic, L.; Skrinjar, R.; Indihar-

stemberger, M., 2008. Organizational

PerformanceMeasures for Business Process

Management: A Performance MeasurementGuideline,

Computer Modeling and Simulation, UKSIM 2008. pp.

94,99, 1-3. April.

Gil, A. C., 1999. Métodos e técnicas de pesquisa social. São

Paulo: Atlas.

Heidari, F.; Loucopoulos, P., 2014. Quality evaluation

framework (QEF): Modeling and evaluating quality of

business processes, International Journal of

Accounting Information Systems, Volume 15, Issue 3,

September, pp. 193-223, ISSN 1467-0895,

Hou, H.; Song, Q.; You, G.; Hao, K., 2009. The research of

Metrics 0repository for Business ProcessMetrics.

China: Second International Symposium on

Computational Intelligence and Design, pp. 168-171.

Kitchenham, B., 2007. Guidelines for performing

Systematic Literature Reviews in Software

Engineering. Reino Unido: EBSE Technical Report,

version2.3.

Leclair, C.; Cullen, A.; Keenan, J., 2012. Use A Metric

Framework To Drive BPM Excellence in: Forrester.

P. kueng, P.; Krahn, A., 1999. Building a Process

Performance Meaasurement System: some Early

experiences, Journal of Scientific and Industrial

Research, National Institute of Science Communication

and Information Resources, New Delhi, Vol. 58, No. 3-

4, pp. 149-159.

Pidun, T., J. Buder and C. Felden, 2011. Optimizing

Process Performance Visibility through Additional

Descriptive Features in Performance Measurement.

International Enterprise Distributed Object Computing

Conference Workshops, pp. 204–212.

Pidun, T.; Felden, C., 2012. Two Cases on How to Improve

the Visibility of Business Process Performance, System

Science (HICSS), 2012 45th Hawaii International

Conferenceon, vol. ??, no., pp.4396-4405, 4-7 Jan.

Powell, SG.; Schwaninger M.; Trimble, C., 2011.

Measurement and control of business process. Syst

Dyn 17:63-91.x.

Škrinjar, R.; Bosiljvukšić, R.; Indiharštemberger, M.

(2008). The impact of business process orientationon

financial and non financial performance, Business

Process Management Journal, Vol. 14 Iss: 5, pp.738-

754.

Smith, G.; Furt, S., 2009. How (not) to Fail at BPM.

BPM.COM. URL: http://bpm.com/how-not-to-fail-at-

bpm.html (visited 07/18/2013).

Trkman, P., 2010. The critical success factors of business

process management. International Journal of

Information Management, v. 30, n. 2, pp. 125-134,

Wang, P.; HE, W., 2012. Research on key performance

indicator (KPI) of business process. China: Second

International Conference on Business Computing and

Global Informatization, pp. 151-154.

Yin, R. Estudo de caso: planejamento e métodos. 3. ed.

Porto Alegre: Bookman, 2005.

A Procedural Approach for Evaluating the Performance of Business Processes

527