A Practical Extension of Frameworks for Auditing with Process Mining

Ella Roubtsova

and Niels Wiersma

Open University of the Netherlands, The Netherlands

Municipality Eindhoven, The Netherlands

Keywords:

Audit, Audit Statements, Compliance Patterns, Frameworks for Audit with Process Mining.

Abstract:

Audit of processes to verify legal compliance is a necessary activity in banks, municipalities and many other

sectors. In theory, by using log-ﬁles and process mining tools, auditors can automate the auditing process

instead of data gathering and taking samples. However, audits are rarely supported by process mining tools in

practice. This paper investigates the reasons for that. We identiﬁed the fact that the published audit frameworks

with process mining are not oriented on real auditors. They replace the auditors with compliance experts and

do not see the necessary steps of reﬁnement of audit statements to make them useful both for writing process

mining ﬁlters and for analysis of the process instances on correspondence to audit goals and policies. We

also identiﬁed that the building or correcting business process models for audit is often driven by the log

information.

On the basis of our ﬁndings, we propose an extension of the audit frameworks with process mining and

evaluate our extension by conducting case studies of audit in different business domains.

1 INTRODUCTION

Auditing is a standard business practice with many

applications. Traditionally, the ﬁrst thing that comes

to mind when thinking about auditing is a ﬁnancial

context, aimed at examining compliance of a business

to tax rules and regulations. This is only one of many

uses. Today, one can audit maintenance engineering

practices, health and safety issues, ethical conduct,

and a wide variety of IT-related practices such as in-

formation systems security and access control (Jans

et al., 2013; Jans et al., 2014; Accorsi and Stocker,

2012).

An audit can be deﬁned as an “independent and

documented system for obtaining and verifying audit

evidence, objectively examining the evidence against

audit criteria, and reporting the audit ﬁndings, while

taking into account audit risk”(Karapetrovic and Will-

born, 2000). Process audit is one of many audit types,

which is aimed at the auditing of a speciﬁed business

process against documented procedures. According

to (Russell, 2006) business process audits are used to

measure conformance to standards and requirements

of the product that is delivered through the process.

Another objective of audit may be measuring the ef-

fectiveness of the process and the instructions that de-

liver the product.

When introducing the research ﬁeld of process mi-

ning, (van der Aalst, 2011) presented the extracting

of knowledge from event logs of information sys-

tems as an opportunity to discover processes, check

conformance of processes against a predeﬁned model

and enhance models. Recently, the necessary techni-

ques have implemented in the ever-increasing num-

ber of process analysis tools, such as (DISCO, 2016),

(ProM, 2016), (bupaR, 2017) and many others. Still,

process mining is rarely used for audit in practice.

This paper analyses the reasons, identiﬁes the mis-

sing steps and proposes a practical extension for audi-

ting frameworks with process mining. The proposed

extended framework is validated by two case studies

of audit of a grant application process and a process

of handling invoices.

The structure of the paper is the following.

Section 2 describes related work, including process

mining, audit statements, conformance patterns and

existing frameworks for audit with process mining.

This section contains our ﬁndings about possible rea-

sons of rare practical use of process mining for audit

in practice. In Section 3 we propose a practical ex-

tension for auditing frameworks with process mining.

Section 4 contains results of two case studies aimed at

evaluating the proposed extension. Section 5 discus-

ses the results of evaluation of the proposed extension

406

Roubtsova, E. and Wiersma, N.

A Practical Extension of Frameworks for Auditing with Process Mining.

DOI: 10.5220/0006798904060415

In Proceedings of the 13th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2018), pages 406-415

ISBN: 978-989-758-300-1

and invites applying the proposed extension for exis-

ting frameworks for future work.

2 RELATED WORK

2.1 Process Mining

Process mining is deﬁned as the activity of discove-

ring, monitoring and improving real processes by ex-

tracting knowledge from event logs that are present in

Information Systems (van der Aalst, 2012).

Central to process mining is an event log, which

contains log entries of events that are captured by an

information system.

Each entry of a log presents an event and consists

of at least the following information: (case designa-

tion, activity label, time stamp). In practice, logs may

contain more information; so an event can be presen-

ted as an instance of a tuple (case designation, acti-

vity label, time stamp, resource, performer, product

description, order size...).

An event log of a process can be seen as a record

of all events of all cases of the process within a certain

time interval. Each case is a sequence of events.

There are different types of process mi-

ning (van der Aalst, 2011):

• Process mining may be organized as a process dis-

covery from a log. “For example, well-known al-

gorithms such as the Alpha algorithm can auto-

matically extract a Petri net that gives a concise

model of the behavior seen in the event log. This

gives the auditor an unbiased view on what has ac-

tually happened” (van der Aalst et al., 2010). The

model extracted from a log may be too detailed

for the level of abstraction needed to the auditor.

• Process mining may be organized as model con-

formance checking that uses a predeﬁned process

model and compares this model with the data in

the event log. By doing this, one can answer que-

stions regarding conformance of a real-world pro-

cess as recorded in the event log to the model of

the process as it should be. A predeﬁned model is

a prerequisite for conformance checking.

• Process mining may validate the compliance of a

logged process with the rules speciﬁed for a gi-

ven business process. Compliance with various

laws, regulations and standards is a well known

problem in business process development and ma-

nagement. The compliance can be checked at the

design time (Awad et al., 2009) and at run-time

(Barnawi et al., 2016).

2.2 Process Audit, Audit Statements

Such process mining types as process discovery and

conformance checking do not completely correspond

to the process audit in its traditional sense.

Process audit in practice is deﬁned as obtaining

evidence that a process is in compliance with predeﬁ-

ned rules called audit statements. Practical audit usu-

ally does not demand the existence of a given busi-

ness process model, as some adhocracy and adaptivity

are considered acceptable (Mintzberg and McHugh,

1985). Practical audit is executed unexpectedly or pe-

riodically. It’s major goal is to ﬁnd evidence of viola-

tions of predeﬁned rules and possibly ﬁnd the reasons

of violations (Karapetrovic and Willborn, 2000). The

audit evidence is often obtained manually, by con-

ducting interviews with user about the process they

follow, or taking samples of cases that are executed

in the system. This evidence must then be compa-

red against documented procedures. The documented

procedures can be presented as a number of written

rules and regulations called audit criteria that pertain

to the business process and should be observed while

executing it (Karapetrovic and Willborn, 2000). In ot-

her words, the whole business process is usually not

speciﬁed in the documented procedures of audit.

Process audit risk inﬂuences the scope of audit

statements. The auditor focuses his or her effort on

formulating audit statements in the areas where most

risk is perceived. The risk is calculated as a function

depending upon the number of cases (process instan-

ces) that non-compliant with the audit statement, and

as the ﬁnancial loss associated with all non-compliant

cases.

Audit statements specify the important rules as

principles which are not ready for process mining as

they do not name the process activities and data items

in the log structure.

For example, a simple ordering based rule is spe-

ciﬁed as ”no change to a request can be made after it

is approved” (van der Aalst, 2011). In order to auto-

mate the compliance checks to this rule, the auditor

needs some knowledge or assumption about the bu-

siness process activities that can be found in the log.

She can assume that there are activities ”Order Ap-

proved” and ”Request Order Change”. The auditor

should see a case as a sequence of events where the

activities have ordering relations, say one activity can

follow another. If these assumptions are made, then

the audit statement can be formulated as: ”Activity

Order Approved must never be followed by activity

Request Order Change (for the same Order)”.

A Practical Extension of Frameworks for Auditing with Process Mining

407

2.3 Compliance Patterns

Audit statements are the rules that should be checked

for the business process under the audit, so the classi-

ﬁcation of rules as process compliance patterns used

by the compliance checking (Ly et al., 2013) is appli-

cable for audit.

The compliance patterns have been conceptually

presented in (Barnawi et al., 2016). A compliance

pattern is ”an abstract speciﬁcation of monitoring re-

quirements and it covers the major structural facets of

business processes: (1) Occurrence, (2) Order (with

or without time span) and (3) employed resources ”

(Barnawi et al., 2016).

The group of Occurrence Patterns presents Exis-

tence or Absence of activities with given values of

process data.

The group of Order Patterns often includes the

time span information and presents a sequence of acti-

ons often with time stamps to deﬁne a time span, a

precedence or a response.

The group of Resource Patterns presents Binding

Of Duty, Segregation of Duty or Responsibility (Per-

formed By). This group is often called Agent Based.

There is a group of Product Patterns that names

the states of the products handled or produced by the

business process.

The compliance patterns can be logically compo-

sed. In general, a process compliance pattern is an ex-

pression supported with the temporal logic that infor-

mally means ”there are (or there are no) cases where

an Order Pattern deﬁned on activities is met, and(or)

the Time Span between activities is within the norm,

and(or) the given Resource was used and(or) by the

given Role, and(or) the speciﬁed Product is produced

(or ordered) ...” (Roubtsova, 2005).

The concept of anti-patterns (pattern negations, or

negation of sub-expressions of a pattern) can be also

used as the process instances that violate the rules are

the target of audit (Barnawi et al., 2016).

Compliance patterns are implemented in process

mining tools as log-ﬁlters.

The role of an audit statement in audit with pro-

cess mining is twofold. On the one hand, an audit

statement is an instantiation of a compliance pattern

or a composition of patterns. On the other hand, an

audit statement is a means of communication of ex-

perts with different backgrounds. So, the replacing

the formal expression of audit statements with ex-

pressions in a Controlled Natural Language (Spreeu-

wenberg and Healy, 2009) should deﬁnitely support

analysis of audit results. A Controlled Natural Lan-

guage preserves the terminology used by auditors and

business process experts and preserves the structure

of compliance patterns in expressions of audit state-

ments.

2.4 Existing Frameworks for Audit with

Process Mining

The existing frameworks for audit with process mi-

ning assume that the business process model is given

or is built for the audit. It is assumed that the audit

statements are ready for process mining.

Indeed, the framework presented by (Sadiq et al.,

2007) suggests to ﬁnd a business process model in

the log and check the compliance of the logged pro-

cess with audit statements. The straightforward ap-

plication of this framework is problematic. The real-

world process instances in the event log may not con-

tain the activities and other data mentioned in audit

statements.

The framework (van der Aalst et al., 2010) is prin-

cipally designed to mine and compare de-jure and de-

facto business process models.

The BP-MaaS framework (Barnawi et al., 2016)

(see Figure 1) for run-time compliance checking sup-

poses that “Business process management practice

commences with the Business Expert deﬁning

and modelling business process requirements using

BPMN. ...The BPMN model follows multiple itera-

tions of design and reﬁnement to faithfully represent

business logic and requirements. The outcome is a

BPMN model capturing the control and the dataﬂow

of the business logic.” The BPMN model is used

to ﬁlter only cases of the audited process from logs.

Another role, is the role of Compliance Expert, who

is responsible for formulating audit statements using

the compliance patterns.

We may evidence the presence of Business Ex-

perts in the practice of iterative building of a process

model suitable for audit. However, we do not agree

that each company has a Compliance Expert. The tra-

ditional role of an Auditor, who formulates audit sta-

tements as principles and may be not acquainted with

compliance patterns and details of the business pro-

cess, is not present within the BP-MaaS framework.

We observed the roles of an Audit Expert, and an

IT Expert involved in organizing an audit with process

mining. This observation has led us to a proposal of

an extension for frameworks for audit with process

mining.

ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering

408

Complience Mangement Knowledge Base

Business

Expert

Compliance

Expert

Business Process

Model

Execution Engine

Logs

Compliance Rule

Editor

Automated mapping

into target rules/

queries

Monitoring based

on anti patterns

Dashboard

Cases that do not

meet a target rule

Figure 1: BP-MaaS framework for run-time compliance

checking.

3 AN EXTENSION FOR

FRAMEWORKS FOR AUDIT

WITH PROCESS MINING

The practice shows that the auditors see the audited

process abstractly, see only important principles. The

auditors often use different terminology than the ter-

minology of business process experts. Audit state-

ments almost always need reﬁnement to the process

activities existing in a business process and recorded

in logs.

However, the logs usually record activities of

many business processes. Moreover, the names of

activities in the logs may deviate from the names used

by business experts (for example, they may be abbre-

viated). Sometimes one business process is recorded

in different logs. So, the logs need to be prepared and

ﬁltered for the audited business process.

The research question of this work is the follo-

wing: What steps should be included into the frame-

works for audit with process mining to relate the audit

statements formulated by Audit Experts, the business

process known by Business Experts, and the logged

process activities available for IT Experts, and ena-

ble audit with process mining? We have found in

the literature a recommendation that “A participatory

Log

Normative Business Process

Model (uses the names of

activities existing in the Log)

Audit

statements

Documents about

business process

Audit statements written using

Controlled Natural language

(uses names of activities and

fields existing in the log)

Participatory workshop

IT Expert, Business Expert, Audit Expert

Log filtered against the

Normative Business Process

Model

Process mining

IT Expert

Audit with process mining

Filters corresponding to audit

statements

Analysis of filtered cases

IT Expert, Business Expert and Audit Expert

Cases that do or don’t meet

the audit criteria

Figure 2: Our Extension for Frameworks for Audit with

Process Mining.

modeling workshop is appropriate for the situations

where the facilitated modeling with a group of stake-

holders is crucial for the implementation of new bu-

siness technology. It is particularly useful when an

agreement and solution can be completely covered if

all stakeholders participate in discussion” (Sandkuhl

et al., 2014).

We propose to extend the frameworks for audit

with process mining with a participatory workshop

to relate terminology of three groups of experts. Fi-

gure 2 shows our extension for audit with process mi-

ning with a participatory workshop. The expected re-

sults of the workshop are a Normative Business Pro-

cess Model and a set of Audit Statements written in

a Controlled Natural Language. The process and the

statements should use names of activities and ﬁelds

existing in the log.

In the next section, we report the evidence of the

necessity and usefulness of such an extension in two

case studies of audit with process mining in different

business domains.

A Practical Extension of Frameworks for Auditing with Process Mining

409

4 EVALUATION OF THE

PROPOSED EXTENSION

We have conducted two case studies in different busi-

ness domains in order to show evidence that

• the audit criteria formulated by Audit Experts

need reﬁnement allowing one to relate them to the

process activities;

• the event logs often do not contain the names of

activities used by Business Experts for process

description;

• audit with process mining needs a participatory

workshop combining an IT Expert, a Business Ex-

pert and an Audit Expert. The workshop should

create a model of the Normative Business Pro-

cess with the names of activities used in the log.

The workshop should reformulate the Audit state-

ments in a Controlled Natural Language that uses

the names of activities from the Normative Busi-

ness Process, and phrases indicating ordering of

activities, resources and other concerns used in

compliance patterns.

The validation of the results of a participatory

workshop is presented as the number of audit sta-

tements formulated after the proposed steps and

checked with process mining techniques.

4.1 Audit of a Grant Application

Process

The process description found in documentation was

rather informal.

In the process of grant applications, citizens or lo-

cal institutions apply for a monetary allowance pro-

vided by the city. This is called a grant, and can be

used to organize an (yearly or one-time) activity that

contributes to the communal goals that the city has

deﬁned. Grants can be awarded for several small-

scale goals, such as sporting events and local festivals,

but also large grants for welfare support of special-

interest groups are issued.

A grant application should be sent within the de-

adline. After that it is examined, additional informa-

tion may be requested. All steps of the application and

examination should be done in predeﬁned time slots.

After all the examinations, a grant can be awarded or

the application can be rejected.

4.1.1 Initial Log

To obtain the log, an IT Expert exported the log, trans-

lated metadata, anonymized data and imported the log

into the process mining application. The log was then

ﬁltered for year of grant 2014, as this was fully contai-

ned in the exported time period of 2011 − 2015. The

csv-ﬁle (comma-separated values) of the log was lo-

aded as a spreadsheet. There were found 132 unique

events (activities) that are part of the grant applica-

tion process. The log contains the following ﬁelds:

(1) Case Number, (2) Case Description, (3) Year of

Grant, (4) Amount requested, (5) Amount granted, (6)

Grant Regime, (7) Grant Type, (8) Date of payment,

(9) Activity name, (10) Resource name (anonymized),

(11) Activity date(time).

The deviation of the informal process description

from the log structure is already recognized. For ex-

ample, it is not clear what ”Grant Type” and “Grant

Regime” mean. The process description does not con-

tain names of process activities.

4.1.2 Control Objectives and Audit Statements

We have observed, that an Audit Expert begins his

work with control objectives. A control objective is a

generic statement that is applicable on the entire dom-

ain of a law or regulation. Among such objectives are

respecting the deadlines, classiﬁcation of cases, prio-

ritizing of particular properties of a process case.

In order to support process mining, any control

objective should be formulated in terms of the audi-

ted business process. However, the business process

was informally deﬁned. So, the ﬁrst versions of 52

audit statements found in documents were also rather

informal. We present examples of the initial audit sta-

tements.

• ST 1 : Every grant application is required to be submit-

ted on time.

• ST 13 : The activities that will be executed if the grant

is approved must be described in a SMART manner.

• ST 14 : During the grant approval process, the ﬁnancial

review and content review cannot be performed by the

same employee.

• ST 25 : If the grant amount that is approved is larger than

50.000EU R, a semi-annual progress report is required

by July, 1st at the latest.

Analyzing these audit criteria, one can think of

activities that can be presented in the log, like sub-

mission, approval, review, progress report. However,

one can never be sure, if these names of activities are

present in the business process and the log. An agre-

ment on a “normative process” should be achieved in

the organization between business process experts, IT

experts and auditors, so that the audit can be fulﬁlled

and its results can be interpreted.

The facts presented in sections 4.1.1 and 4.1.2

show the evidence that a participatory workshop was

needed.

ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering

410

4.1.3 Normative Process as a Result of a

Participatory Workshop

A participatory workshop was organized. The ﬁrst

purpose of the workshop was an agreement on a nor-

mative process, containing the activities found in the

log.

First, the experts agreed on the grant application

process shown in Figure 3. It contains 11 activities

and the end-state of grant allocation.

However, during the workshop, the Business and

IT Experts identiﬁed that each decision point of the

normative process shown in Figure 3 in reality has

more outcomes. A grant may be approved, rejected,

partially approved, approved with a sanction etc.

When the real normative outcomes were listed, the

normative process recoded in the log was ﬁltered with

the help of the Disco tool. It contains 56 activities

(see Figure 4). In the Disco tool, the colour intensity

of the activity corresponds to the frequency of its use

in cases (process instances). So, the grey activities are

used in a smaler number of cases than the dark blue

activities. The names of activities and the frequen-

cies of their use were zoomed in and the list of real

activities was corrected.

4.1.4 Audit Statements in Controlled Natural

Language (CNL)

Using the names of activities found in the log and the

ordering of activities in the normative business pro-

cess, the 52 audit statements were revised and refor-

mulated to be used in the process mining tool. We

present examples of audit statements. All statements

can be found in (Wiersma, 2017).

For example,

ST1 (informal): Every grant application is required

to be submitted on time.

got the new formulation:

ST 1(CNL) : For every case where the grant regime

equals “average” or “large”, the timestamp of acti-

vity “proposal received” must be earlier than 31-10-

2013.

Another example:

ST7 (informal): If a request to supply missing infor-

mation has been sent to the applicant, the missing in-

formation need to be provided by the applicant within

10 working days.

ST7 (CNL): For every case if the activity “DI datum

request missing information” is executed, the time be-

tween this activity and the activity “Receipt of missing

information” must be < 14 days (the weekends have

been added).

There is a statement corresponding to the the so-

called four-eye principle.

ST48 (informal): During the grant application pro-

cess, the ﬁnancial review and content review cannot

be performed by the same employee.

ST48 (CNL): For all cases where activities “review

context” and “review ﬁnancial part” must be revie-

wed by two different resources.

Some audit criteria specify the details of speciﬁc

cases. For example,

ST36 (informal): Grants > 100.000 must submit an

accountant statement.

ST36 (CNL): For all cases where the attribute

“amount received”> 100.000, activity “additional

information received” must contain an accountant

statement.

Some audit criteria were used to correct the nor-

mative process. For example, it was found that two

reviews (context and ﬁnance) were required to be per-

formed in the real process.

S11: Every application is reviewed with respect to

content by the grant expert. The review is documen-

ted.

S12: Every application is reviewed with respect to ﬁ-

nance by the grant account manager. The review is

documented.

There were initial audit statements containing

events (activities) that were not present in the log. For

example:

ST8: For every case where activity “Request additio-

nal information” is performed, a notiﬁcation of sus-

pension is included in the letter. All letters that can be

sent to grant applicants are digitally available, and

can be checked for inclusion of the suspension notice.

It was found that letters remain a human one-time

activity and not reﬂected in the log.

There were audit statements that could be seen as

principles for a group of other audit statements. For

example, “ST47: The activities that will be execu-

ted if the grant is approved, must be described in a

SMART manner”. ST47 is a principle for a group of

audit statements “ that will be executed if the grant

is approved” to be Speciﬁc, Measurable, Assignable,

Realistic and Time-related.

4.1.5 Results of the Audit with Process Mining

In total, 27 statements out of 52 were made suitable

for process mining. This shows the degree of audit

support that can be achieved with process mining. In

case of the design of logs of processes for automated

audit, the degree of support can be increased.

Results of the audit with process mining are the

following. 23 of 27 ﬁlters of all veriﬁable audit state-

ments did have one or more non-compliant cases that

were included in the event log. In the Disco-tool, the

A Practical Extension of Frameworks for Auditing with Process Mining

411

Normative Process Model

Grant Application Process

Management Advisor Administrative

1. Open

Case File

3. Request

missing

information

2. Send

Confirmation

Letter

Start Process-

Application

received

5. Request

additional

information

6. Prepare

Decision

4. Review

Application

Complete?

yes

Clarification

needed?

7.Conform

Decision

8. Send

Acceptance/

Rejection letter

9. Send

request for

justification

Justification needed?

10. Sanction

11. Evaluate

Justification

On time?

12. Definitive

grant

allocation

yes

Figure 3: Initial Normative process.

Figure 4: Corrected grant application process.

non-compliant cases can be visually identiﬁed by ap-

plying the appropriate ﬁlter for the audit statement.

For example, after application of the ST 7 ﬁlter

(about deadlines for receiving missing information),

the process map, containing four cases, can be analy-

zed to ﬁnd the reasons of non-compliance (Wiersma,

2017).

ST 7 has four cases that are non-compliant. Is

this good or bad? This depends on the risks associ-

ated with compliance to this statement. The analysis

should be done by the Audit and Business Experts.

4.1.6 Results of the Case Study

The audit of the Grant Application Process has shown

evidence that a participatory workshop is the key step

to the success of frameworks for audit with process

mining.

In the participatory workshop, the business model

described by business experts was corrected by con-

fronting with the activities recorded in the log. Audit

statements were rewritten using the names of activi-

ties in the log, and the norms used in the business

process.

After such preparations, the process mining was

used for compliance checks of the business process to

the audit statements.

4.2 Audit of a Process of Handling

Invoices

The second case study was conducted in a ﬁnan-

cial department, because the processes in this dom-

ain are a subject to law regulations and have guidance

proscribed by the national professional accountants

organizations, like, for example, (PAiE, 2017).

The question was if the identiﬁed need for a parti-

cipatory workshop depends on the domain of audit.

ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering

412

4.2.1 Evidence of the Need of a Participatory

Workshop

An organization uses services and purchases goods.

The service and goods providers send invoices as re-

quests for payment for certain goods or services. The

handling of purchasing invoices is a ﬁnancial process.

We have found the following sources with some

elements of the process model: (a) a process break-

down in a table of statements that is used by the inter-

nal auditors; (b) a user process instruction for speciﬁc

roles.

We have collected the initial information about

process activities, their relation to business roles and

logs in a table (see Figure 5) to present it during the

participatory workshop. Figure 5 shows the absence

of a well-described business process model of the au-

dited process. The found audit statements were in the

form of control objectives. A participatory workshop

was needed.

4.2.2 Results of a Participatory Workshop

One result of the participatory workshop was a Nor-

mative Process. The limited space for the paper does

not allow us to present it. The process can be found

in (Wiersma, 2017).

Another result is the 55 audit state-

ments (Wiersma, 2017). Following our framework

extension, the statements were rewritten for compli-

ance checks using a Controlled Natural Language.

We faced with the fact that not all 55 statements

were unambiguous. For instance, statement ST5.1 is

ambiguous:

ST5.1. Imported invoices are rerouted to employees

that can select the department for further processing

of the invoice after the necessary information is ad-

ded.

Our sources contained no further information that

could be used to assert which data is considered ne-

cessary.

The participatory workshop has shown that no log

of the Purchasing Invoices Process is available. The

Purchasing Invoices Process workﬂow is implemen-

ted in the OpenText eDocs Document Management

System, which is used by all departments. In addition

to this system, the scanning and processing of recei-

ved invoices was done in a KTM (Kofax Transfor-

mation Module) application. Finally, the actual pay-

ment order to the bank is processed by the ﬁnancial

system. Logging of this system was available for au-

diting, but was too complex to integrate in the audit.

To obtain the event log of the invoices workﬂow, IBM

Cognos BI was used to select and export the data from

the eDocs Document Management System. For this

case, a selection is made based on the time stamps

of case activities in eDocs with date values between

01 − 01 − 2015 and 31 − 12 − 2015.

Even in the found log, not all audit criteria could

be transformed into a ﬁlter of a process mining tool,

because the log did not contain necessary informa-

tion. The ﬁrst two activities of the process (scan-

ning and importing) are not part of the activities in

the event log as these are performed in a different sy-

stem. As a result, we could use only 11 out of 55

audit statements for the process mining and ﬁnding

the compliant and non-compliant cases.

5 RESULTS AND CONCLUSIONS

In this work, we have evaluated the upcoming techno-

logy of audit with process mining.

Analysing the literature, we have found frame-

works for audit with process mining that assume exis-

tence of business experts with the knowledge of a bu-

siness process model, and compliance experts, who

are able to formulate audit statements applicable for

process mining tools.

Conducting the case studies, we have found that

the companies more likely have IT Experts, Audit

Experts and Business Experts. The success of ap-

plication of process mining for audit depends on

agreement on terminology achieved by these three

groups of experts. We assumed that this agreement

can be achieved in a participatory workshop.

We have proposed to include such a workshop in

any practical audit framework with process mining in

order to manage the expectations of businesses, that

plan to apply process mining for audit.

We have applied an extended framework in two

case studies in different business domains and succee-

ded to design the normative business processes and

audit statements that can be used for compliance

checks on logs.

Both case studies have evidenced that the direct

application of process mining for audit is a too large

step that cannot be made by the business without in-

termediate steps of designing the normative business

processes and audit statements applicable for process

mining on the process logs. In our ﬁrst case study, the

business process model provided by business experts

was corrected by extra outputs of decisions. The cor-

rection came from the log analysis. In the second case

study, the business process model was created from a

table of partial descriptions of process steps.

Both of our case studies have evidenced that such

a business role as ”Compliance Expert” does not exist

and the initial audit statements made by Audit Experts

A Practical Extension of Frameworks for Auditing with Process Mining

413

Name

Description

Role

Activity in log

Scan invoice

Incoming invoice (by mail) is scanned as part

of a batch of invoices

Department

Invoices

None (Kofax

system)

Check / complete

scanned invoice

The text on the invoice is automatically

recognized (OCR) and translated to specific

fields: date on the invoice, invoice no., bank

account number, commitment number and

amount payable. The result of the OCR is

visually checked against the digital scan (and

completed / corrected)

Department

Invoices

None (KTM

system)

Send invoice to

DMS

The invoice is exported from KTM and

imported into the Document Management

System (eDocs)

Department

Invoices

Purchasing

Invoices 2.0

Complete invoice

details

The fields required for further handling that

were not imported are completed: invoice

description, tax code and amount,

department to authorise, number of supplier

(looked up in Decade)

Financial data

entry

Purchasing

Invoices 2.0

Invoice at block

manager

The invoice is sent to the block manager for

re-routing to the department

Financial data

entry

1.Invoice at block

manager

Re-route to

department

The block manager re-routes the invoice to

the department for authorization

Block manager

2.Receive and

reroute

Encode invoice

The invoice is re-routed to the right resource

for the performance check, to verify that the

goods that are billed are delivered.

Department

inbox

3.Encode invoice

Correctly Booked?

If the department thinks the invoice is

incorrectly put in their inbox, it is rerouted

back to the block manager.

Performance

check

The performance check is executed by

sending an ‘Ok’ and a comment in the eDocs

system. This step is not enforced and

sometimes skipped.

Checker

4.Performance

check

Encode invoice

The booking details (cost centre and booking

combination) are entered.

Department

inbox

3. Encode invoice

10.

Check booking

details

The invoice is sent to a resource that checks

the booking details. This step can be skipped

Checker

5. Booking entry

check

11.

Approval

The invoice is approved by the budget

administrator (in case of a commitment

already exists in Decade) or the budget

manager. Approval is done for each invoice

line.

Budget admin /

budget mgr.

6. Approve invoice

Authorize payment

/ process invoice

The complete invoice and payment details

are checked and payment is authorised. After

authorisation, the payment is automatically

processed by Decade.

Financial

advisor

Process invoice

Figure 5: Table for the Participatory workshop.

should be rewritten for process mining by an IT expert

in collaboration with an Audit Expert and a Business

Expert.

So, the cases demonstrate usefulness of our pro-

posal to include a participatory workshop into audit

frameworks for audit with process mining. They also

show the types of artifacts that are created by the par-

ticipatory workshop: a Normative Business Process

Model using the activities mentioned in the log and

a set of Audit Statements formulated in a Controlled

Natural Language. A Controlled Natural Language

uses the activities mentioned in the log and the com-

pliance patterns that can be checked in the log by pro-

cess mining tools.

The construct validity of our case studies is gua-

ranteed by the many sources of information used for

conduction of them (triangulation). Indeed, any busi-

ness model and statement was inspected by three dif-

ferent specialists using different documents.

The conduct validity of our case studies can be

checked by everyone as we have presented the steps

of the proposed framework and results for research re-

production, as it is recommenced in literature on case

studies (Yin, 1994). All details about case studies can

be found in (Wiersma, 2017). The logs can be provi-

ded on request.

ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering

414

Although we used the Disco tool for process mi-

ning, our extension is tool independent and can be

applied for the frameworks that work with different

process mining tools.

As future work, we invite using our extension for

audit frameworks with process mining. This may re-

sult in a reusable collection of normative business

processes and audit statements for speciﬁc domains

of audit application.

REFERENCES

Accorsi, R. and Stocker, T. (2012). On the exploitation of

process mining for security audits: the conformance

checking case. In Proceedings of the 27th Annual

ACM Symposium on Applied Computing, pages 1709–

1716. ACM.

Awad, A., Weidlich, M., and Weske, M. (2009). Speci-

ﬁcation, veriﬁcation and explanation of violation for

data aware compliance rules. Service-Oriented Com-

puting, pages 500–515.

Barnawi, A., Awad, A., Elgammal, A., Elshawi, R., Alma-

laise, A., and Sakr, S. (2016). An anti-pattern-based

runtime business process compliance monitoring fra-

mework. framework, 7(2).

bupaR (2017). Business Process Analysis in R.

http://bupar.net/.

DISCO (2016). Flexicon DISCO.

http://ﬂuxicon.com/disco/.

Jans, M., Alles, M., and Vasarhelyi, M. (2013). The case

for process mining in auditing: Sources of value ad-

ded and areas of application. International Journal of

Accounting Information Systems, 14(1):1–20.

Jans, M., Alles, M. G., and Vasarhelyi, M. A. (2014). A

ﬁeld study on the use of process mining of event logs

as an analytical procedure in auditing. The Accounting

Review, 89(5):1751–1773.

Karapetrovic, S. and Willborn, W. (2000). Generic audit

of management systems: fundamentals. Managerial

Auditing Journal, 15(6):279–294.

Ly, L. T., Maggi, F. M., Montali, M., Rinderle-Ma, S., and

van der Aalst, W. M. (2013). A framework for the

systematic comparison and evaluation of compliance

monitoring approaches. In Enterprise Distributed Ob-

ject Computing Conference (EDOC), 2013 17th IEEE

International, pages 7–16. IEEE.

Mintzberg, H. and McHugh, A. (1985). Strategy formation

in an adhocracy. Administrative science quarterly, pa-

ges 160–197.

PAiE (2017). Professional Accountants in Europe.

http://paie.nl.

ProM (2016). Process mining Workbench ProM.

http://www.promtools.org.

Roubtsova, E. E. (2005). Property driven mining in work-

ﬂow logs. In Intelligent Information Processing and

Web Mining, pages 471–475. Springer.

Russell, J. (2006). Process auditing and techniques. Quality

Progress, 39(6):71–74.

Sadiq, S., Governatori, G., and Namiri, K. (2007). Mo-

deling control objectives for business process compli-

ance. Business process management, pages 149–164.

Sandkuhl, K., Stirna, J., Persson, A., and Wißotzki, M.

(2014). Enterprise modeling. Tackling Business Chal-

lenges with the 4EM Method. Springer, 309.

Spreeuwenberg, S. and Healy, K. A. (2009). Sbvrs ap-

proach to controlled natural language. In Internatio-

nal Workshop on Controlled Natural Language, pages

155–169. Springer.

van der Aalst, W., van Hee, K. M., van der Werf, J. M., and

Verdonk, M. (2010). Auditing 2.0: Using process mi-

ning to support tomorrow’s auditor. Computer, 43(3).

van der Aalst, W. M. P. (2011). Process Mining - Discovery,

Conformance and Enhancement of Business Proces-

ses. Springer.

van der Aalst, W. M. P. (2012). Process mining: Overview

and opportunities. ACM Transactions on Management

Information Systems (TMIS), 3(2):7.

Wiersma, N. (2017). The use of Process Mining in Business

Process Auditing, http://hdl.handle.net/1820/7702.

Open University of the Netherlands.

Yin, R. (1994). Case study research: Design and methods .

beverly hills.

A Practical Extension of Frameworks for Auditing with Process Mining

415