A Practical Extension of Frameworks for Auditing with Process Mining
Ella Roubtsova
1
and Niels Wiersma
2
1
Open University of the Netherlands, The Netherlands
2
Municipality Eindhoven, The Netherlands
Keywords:
Audit, Audit Statements, Compliance Patterns, Frameworks for Audit with Process Mining.
Abstract:
Audit of processes to verify legal compliance is a necessary activity in banks, municipalities and many other
sectors. In theory, by using log-files and process mining tools, auditors can automate the auditing process
instead of data gathering and taking samples. However, audits are rarely supported by process mining tools in
practice. This paper investigates the reasons for that. We identified the fact that the published audit frameworks
with process mining are not oriented on real auditors. They replace the auditors with compliance experts and
do not see the necessary steps of refinement of audit statements to make them useful both for writing process
mining filters and for analysis of the process instances on correspondence to audit goals and policies. We
also identified that the building or correcting business process models for audit is often driven by the log
information.
On the basis of our findings, we propose an extension of the audit frameworks with process mining and
evaluate our extension by conducting case studies of audit in different business domains.
1 INTRODUCTION
Auditing is a standard business practice with many
applications. Traditionally, the first thing that comes
to mind when thinking about auditing is a financial
context, aimed at examining compliance of a business
to tax rules and regulations. This is only one of many
uses. Today, one can audit maintenance engineering
practices, health and safety issues, ethical conduct,
and a wide variety of IT-related practices such as in-
formation systems security and access control (Jans
et al., 2013; Jans et al., 2014; Accorsi and Stocker,
2012).
An audit can be defined as an “independent and
documented system for obtaining and verifying audit
evidence, objectively examining the evidence against
audit criteria, and reporting the audit findings, while
taking into account audit risk”(Karapetrovic and Will-
born, 2000). Process audit is one of many audit types,
which is aimed at the auditing of a specified business
process against documented procedures. According
to (Russell, 2006) business process audits are used to
measure conformance to standards and requirements
of the product that is delivered through the process.
Another objective of audit may be measuring the ef-
fectiveness of the process and the instructions that de-
liver the product.
When introducing the research field of process mi-
ning, (van der Aalst, 2011) presented the extracting
of knowledge from event logs of information sys-
tems as an opportunity to discover processes, check
conformance of processes against a predefined model
and enhance models. Recently, the necessary techni-
ques have implemented in the ever-increasing num-
ber of process analysis tools, such as (DISCO, 2016),
(ProM, 2016), (bupaR, 2017) and many others. Still,
process mining is rarely used for audit in practice.
This paper analyses the reasons, identifies the mis-
sing steps and proposes a practical extension for audi-
ting frameworks with process mining. The proposed
extended framework is validated by two case studies
of audit of a grant application process and a process
of handling invoices.
The structure of the paper is the following.
Section 2 describes related work, including process
mining, audit statements, conformance patterns and
existing frameworks for audit with process mining.
This section contains our findings about possible rea-
sons of rare practical use of process mining for audit
in practice. In Section 3 we propose a practical ex-
tension for auditing frameworks with process mining.
Section 4 contains results of two case studies aimed at
evaluating the proposed extension. Section 5 discus-
ses the results of evaluation of the proposed extension
406
Roubtsova, E. and Wiersma, N.
A Practical Extension of Frameworks for Auditing with Process Mining.
DOI: 10.5220/0006798904060415
In Proceedings of the 13th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2018), pages 406-415
ISBN: 978-989-758-300-1
Copyright
c
2019 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
and invites applying the proposed extension for exis-
ting frameworks for future work.
2 RELATED WORK
2.1 Process Mining
Process mining is defined as the activity of discove-
ring, monitoring and improving real processes by ex-
tracting knowledge from event logs that are present in
Information Systems (van der Aalst, 2012).
Central to process mining is an event log, which
contains log entries of events that are captured by an
information system.
Each entry of a log presents an event and consists
of at least the following information: (case designa-
tion, activity label, time stamp). In practice, logs may
contain more information; so an event can be presen-
ted as an instance of a tuple (case designation, acti-
vity label, time stamp, resource, performer, product
description, order size...).
An event log of a process can be seen as a record
of all events of all cases of the process within a certain
time interval. Each case is a sequence of events.
There are different types of process mi-
ning (van der Aalst, 2011):
Process mining may be organized as a process dis-
covery from a log. “For example, well-known al-
gorithms such as the Alpha algorithm can auto-
matically extract a Petri net that gives a concise
model of the behavior seen in the event log. This
gives the auditor an unbiased view on what has ac-
tually happened” (van der Aalst et al., 2010). The
model extracted from a log may be too detailed
for the level of abstraction needed to the auditor.
Process mining may be organized as model con-
formance checking that uses a predefined process
model and compares this model with the data in
the event log. By doing this, one can answer que-
stions regarding conformance of a real-world pro-
cess as recorded in the event log to the model of
the process as it should be. A predefined model is
a prerequisite for conformance checking.
Process mining may validate the compliance of a
logged process with the rules specified for a gi-
ven business process. Compliance with various
laws, regulations and standards is a well known
problem in business process development and ma-
nagement. The compliance can be checked at the
design time (Awad et al., 2009) and at run-time
(Barnawi et al., 2016).
2.2 Process Audit, Audit Statements
Such process mining types as process discovery and
conformance checking do not completely correspond
to the process audit in its traditional sense.
Process audit in practice is defined as obtaining
evidence that a process is in compliance with predefi-
ned rules called audit statements. Practical audit usu-
ally does not demand the existence of a given busi-
ness process model, as some adhocracy and adaptivity
are considered acceptable (Mintzberg and McHugh,
1985). Practical audit is executed unexpectedly or pe-
riodically. It’s major goal is to find evidence of viola-
tions of predefined rules and possibly find the reasons
of violations (Karapetrovic and Willborn, 2000). The
audit evidence is often obtained manually, by con-
ducting interviews with user about the process they
follow, or taking samples of cases that are executed
in the system. This evidence must then be compa-
red against documented procedures. The documented
procedures can be presented as a number of written
rules and regulations called audit criteria that pertain
to the business process and should be observed while
executing it (Karapetrovic and Willborn, 2000). In ot-
her words, the whole business process is usually not
specified in the documented procedures of audit.
Process audit risk influences the scope of audit
statements. The auditor focuses his or her effort on
formulating audit statements in the areas where most
risk is perceived. The risk is calculated as a function
depending upon the number of cases (process instan-
ces) that non-compliant with the audit statement, and
as the financial loss associated with all non-compliant
cases.
Audit statements specify the important rules as
principles which are not ready for process mining as
they do not name the process activities and data items
in the log structure.
For example, a simple ordering based rule is spe-
cified as ”no change to a request can be made after it
is approved” (van der Aalst, 2011). In order to auto-
mate the compliance checks to this rule, the auditor
needs some knowledge or assumption about the bu-
siness process activities that can be found in the log.
She can assume that there are activities ”Order Ap-
proved” and ”Request Order Change”. The auditor
should see a case as a sequence of events where the
activities have ordering relations, say one activity can
follow another. If these assumptions are made, then
the audit statement can be formulated as: ”Activity
Order Approved must never be followed by activity
Request Order Change (for the same Order)”.
A Practical Extension of Frameworks for Auditing with Process Mining
407
2.3 Compliance Patterns
Audit statements are the rules that should be checked
for the business process under the audit, so the classi-
fication of rules as process compliance patterns used
by the compliance checking (Ly et al., 2013) is appli-
cable for audit.
The compliance patterns have been conceptually
presented in (Barnawi et al., 2016). A compliance
pattern is ”an abstract specification of monitoring re-
quirements and it covers the major structural facets of
business processes: (1) Occurrence, (2) Order (with
or without time span) and (3) employed resources
(Barnawi et al., 2016).
The group of Occurrence Patterns presents Exis-
tence or Absence of activities with given values of
process data.
The group of Order Patterns often includes the
time span information and presents a sequence of acti-
ons often with time stamps to define a time span, a
precedence or a response.
The group of Resource Patterns presents Binding
Of Duty, Segregation of Duty or Responsibility (Per-
formed By). This group is often called Agent Based.
There is a group of Product Patterns that names
the states of the products handled or produced by the
business process.
The compliance patterns can be logically compo-
sed. In general, a process compliance pattern is an ex-
pression supported with the temporal logic that infor-
mally means ”there are (or there are no) cases where
an Order Pattern defined on activities is met, and(or)
the Time Span between activities is within the norm,
and(or) the given Resource was used and(or) by the
given Role, and(or) the specified Product is produced
(or ordered) ...” (Roubtsova, 2005).
The concept of anti-patterns (pattern negations, or
negation of sub-expressions of a pattern) can be also
used as the process instances that violate the rules are
the target of audit (Barnawi et al., 2016).
Compliance patterns are implemented in process
mining tools as log-filters.
The role of an audit statement in audit with pro-
cess mining is twofold. On the one hand, an audit
statement is an instantiation of a compliance pattern
or a composition of patterns. On the other hand, an
audit statement is a means of communication of ex-
perts with different backgrounds. So, the replacing
the formal expression of audit statements with ex-
pressions in a Controlled Natural Language (Spreeu-
wenberg and Healy, 2009) should definitely support
analysis of audit results. A Controlled Natural Lan-
guage preserves the terminology used by auditors and
business process experts and preserves the structure
of compliance patterns in expressions of audit state-
ments.
2.4 Existing Frameworks for Audit with
Process Mining
The existing frameworks for audit with process mi-
ning assume that the business process model is given
or is built for the audit. It is assumed that the audit
statements are ready for process mining.
Indeed, the framework presented by (Sadiq et al.,
2007) suggests to find a business process model in
the log and check the compliance of the logged pro-
cess with audit statements. The straightforward ap-
plication of this framework is problematic. The real-
world process instances in the event log may not con-
tain the activities and other data mentioned in audit
statements.
The framework (van der Aalst et al., 2010) is prin-
cipally designed to mine and compare de-jure and de-
facto business process models.
The BP-MaaS framework (Barnawi et al., 2016)
(see Figure 1) for run-time compliance checking sup-
poses that “Business process management practice
commences with the Business Expert defining
and modelling business process requirements using
BPMN. ...The BPMN model follows multiple itera-
tions of design and refinement to faithfully represent
business logic and requirements. The outcome is a
BPMN model capturing the control and the dataflow
of the business logic. The BPMN model is used
to filter only cases of the audited process from logs.
Another role, is the role of Compliance Expert, who
is responsible for formulating audit statements using
the compliance patterns.
We may evidence the presence of Business Ex-
perts in the practice of iterative building of a process
model suitable for audit. However, we do not agree
that each company has a Compliance Expert. The tra-
ditional role of an Auditor, who formulates audit sta-
tements as principles and may be not acquainted with
compliance patterns and details of the business pro-
cess, is not present within the BP-MaaS framework.
We observed the roles of an Audit Expert, and an
IT Expert involved in organizing an audit with process
mining. This observation has led us to a proposal of
an extension for frameworks for audit with process
mining.
ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering
408
Complience Mangement Knowledge Base
Business
Expert
Compliance
Expert
Business Process
Model
Execution Engine
Logs
Compliance Rule
Editor
Automated mapping
into target rules/
queries
Monitoring based
on anti patterns
Dashboard
Cases that do not
meet a target rule
Figure 1: BP-MaaS framework for run-time compliance
checking.
3 AN EXTENSION FOR
FRAMEWORKS FOR AUDIT
WITH PROCESS MINING
The practice shows that the auditors see the audited
process abstractly, see only important principles. The
auditors often use different terminology than the ter-
minology of business process experts. Audit state-
ments almost always need refinement to the process
activities existing in a business process and recorded
in logs.
However, the logs usually record activities of
many business processes. Moreover, the names of
activities in the logs may deviate from the names used
by business experts (for example, they may be abbre-
viated). Sometimes one business process is recorded
in different logs. So, the logs need to be prepared and
filtered for the audited business process.
The research question of this work is the follo-
wing: What steps should be included into the frame-
works for audit with process mining to relate the audit
statements formulated by Audit Experts, the business
process known by Business Experts, and the logged
process activities available for IT Experts, and ena-
ble audit with process mining? We have found in
the literature a recommendation that A participatory
Log
Normative Business Process
Model (uses the names of
activities existing in the Log)
Audit
statements
Documents about
business process
Audit statements written using
Controlled Natural language
(uses names of activities and
fields existing in the log)
Participatory workshop
IT Expert, Business Expert, Audit Expert
Log filtered against the
Normative Business Process
Model
Process mining
IT Expert
Audit with process mining
Filters corresponding to audit
statements
Analysis of filtered cases
IT Expert, Business Expert and Audit Expert
Cases that do or don’t meet
the audit criteria
Figure 2: Our Extension for Frameworks for Audit with
Process Mining.
modeling workshop is appropriate for the situations
where the facilitated modeling with a group of stake-
holders is crucial for the implementation of new bu-
siness technology. It is particularly useful when an
agreement and solution can be completely covered if
all stakeholders participate in discussion” (Sandkuhl
et al., 2014).
We propose to extend the frameworks for audit
with process mining with a participatory workshop
to relate terminology of three groups of experts. Fi-
gure 2 shows our extension for audit with process mi-
ning with a participatory workshop. The expected re-
sults of the workshop are a Normative Business Pro-
cess Model and a set of Audit Statements written in
a Controlled Natural Language. The process and the
statements should use names of activities and fields
existing in the log.
In the next section, we report the evidence of the
necessity and usefulness of such an extension in two
case studies of audit with process mining in different
business domains.
A Practical Extension of Frameworks for Auditing with Process Mining
409
4 EVALUATION OF THE
PROPOSED EXTENSION
We have conducted two case studies in different busi-
ness domains in order to show evidence that
the audit criteria formulated by Audit Experts
need refinement allowing one to relate them to the
process activities;
the event logs often do not contain the names of
activities used by Business Experts for process
description;
audit with process mining needs a participatory
workshop combining an IT Expert, a Business Ex-
pert and an Audit Expert. The workshop should
create a model of the Normative Business Pro-
cess with the names of activities used in the log.
The workshop should reformulate the Audit state-
ments in a Controlled Natural Language that uses
the names of activities from the Normative Busi-
ness Process, and phrases indicating ordering of
activities, resources and other concerns used in
compliance patterns.
The validation of the results of a participatory
workshop is presented as the number of audit sta-
tements formulated after the proposed steps and
checked with process mining techniques.
4.1 Audit of a Grant Application
Process
The process description found in documentation was
rather informal.
In the process of grant applications, citizens or lo-
cal institutions apply for a monetary allowance pro-
vided by the city. This is called a grant, and can be
used to organize an (yearly or one-time) activity that
contributes to the communal goals that the city has
defined. Grants can be awarded for several small-
scale goals, such as sporting events and local festivals,
but also large grants for welfare support of special-
interest groups are issued.
A grant application should be sent within the de-
adline. After that it is examined, additional informa-
tion may be requested. All steps of the application and
examination should be done in predefined time slots.
After all the examinations, a grant can be awarded or
the application can be rejected.
4.1.1 Initial Log
To obtain the log, an IT Expert exported the log, trans-
lated metadata, anonymized data and imported the log
into the process mining application. The log was then
filtered for year of grant 2014, as this was fully contai-
ned in the exported time period of 2011 2015. The
csv-file (comma-separated values) of the log was lo-
aded as a spreadsheet. There were found 132 unique
events (activities) that are part of the grant applica-
tion process. The log contains the following fields:
(1) Case Number, (2) Case Description, (3) Year of
Grant, (4) Amount requested, (5) Amount granted, (6)
Grant Regime, (7) Grant Type, (8) Date of payment,
(9) Activity name, (10) Resource name (anonymized),
(11) Activity date(time).
The deviation of the informal process description
from the log structure is already recognized. For ex-
ample, it is not clear what ”Grant Type” and “Grant
Regime” mean. The process description does not con-
tain names of process activities.
4.1.2 Control Objectives and Audit Statements
We have observed, that an Audit Expert begins his
work with control objectives. A control objective is a
generic statement that is applicable on the entire dom-
ain of a law or regulation. Among such objectives are
respecting the deadlines, classification of cases, prio-
ritizing of particular properties of a process case.
In order to support process mining, any control
objective should be formulated in terms of the audi-
ted business process. However, the business process
was informally defined. So, the first versions of 52
audit statements found in documents were also rather
informal. We present examples of the initial audit sta-
tements.
ST 1 : Every grant application is required to be submit-
ted on time.
ST 13 : The activities that will be executed if the grant
is approved must be described in a SMART manner.
ST 14 : During the grant approval process, the financial
review and content review cannot be performed by the
same employee.
ST 25 : If the grant amount that is approved is larger than
50.000EU R, a semi-annual progress report is required
by July, 1st at the latest.
Analyzing these audit criteria, one can think of
activities that can be presented in the log, like sub-
mission, approval, review, progress report. However,
one can never be sure, if these names of activities are
present in the business process and the log. An agre-
ment on a “normative process” should be achieved in
the organization between business process experts, IT
experts and auditors, so that the audit can be fulfilled
and its results can be interpreted.
The facts presented in sections 4.1.1 and 4.1.2
show the evidence that a participatory workshop was
needed.
ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering
410
4.1.3 Normative Process as a Result of a
Participatory Workshop
A participatory workshop was organized. The first
purpose of the workshop was an agreement on a nor-
mative process, containing the activities found in the
log.
First, the experts agreed on the grant application
process shown in Figure 3. It contains 11 activities
and the end-state of grant allocation.
However, during the workshop, the Business and
IT Experts identified that each decision point of the
normative process shown in Figure 3 in reality has
more outcomes. A grant may be approved, rejected,
partially approved, approved with a sanction etc.
When the real normative outcomes were listed, the
normative process recoded in the log was filtered with
the help of the Disco tool. It contains 56 activities
(see Figure 4). In the Disco tool, the colour intensity
of the activity corresponds to the frequency of its use
in cases (process instances). So, the grey activities are
used in a smaler number of cases than the dark blue
activities. The names of activities and the frequen-
cies of their use were zoomed in and the list of real
activities was corrected.
4.1.4 Audit Statements in Controlled Natural
Language (CNL)
Using the names of activities found in the log and the
ordering of activities in the normative business pro-
cess, the 52 audit statements were revised and refor-
mulated to be used in the process mining tool. We
present examples of audit statements. All statements
can be found in (Wiersma, 2017).
For example,
ST1 (informal): Every grant application is required
to be submitted on time.
got the new formulation:
ST 1(CNL) : For every case where the grant regime
equals “average” or “large”, the timestamp of acti-
vity “proposal received” must be earlier than 31-10-
2013.
Another example:
ST7 (informal): If a request to supply missing infor-
mation has been sent to the applicant, the missing in-
formation need to be provided by the applicant within
10 working days.
ST7 (CNL): For every case if the activity “DI datum
request missing information” is executed, the time be-
tween this activity and the activity “Receipt of missing
information” must be < 14 days (the weekends have
been added).
There is a statement corresponding to the the so-
called four-eye principle.
ST48 (informal): During the grant application pro-
cess, the financial review and content review cannot
be performed by the same employee.
ST48 (CNL): For all cases where activities “review
context” and “review financial part” must be revie-
wed by two different resources.
Some audit criteria specify the details of specific
cases. For example,
ST36 (informal): Grants > 100.000 must submit an
accountant statement.
ST36 (CNL): For all cases where the attribute
“amount received”> 100.000, activity “additional
information received” must contain an accountant
statement.
Some audit criteria were used to correct the nor-
mative process. For example, it was found that two
reviews (context and finance) were required to be per-
formed in the real process.
S11: Every application is reviewed with respect to
content by the grant expert. The review is documen-
ted.
S12: Every application is reviewed with respect to fi-
nance by the grant account manager. The review is
documented.
There were initial audit statements containing
events (activities) that were not present in the log. For
example:
ST8: For every case where activity “Request additio-
nal information” is performed, a notification of sus-
pension is included in the letter. All letters that can be
sent to grant applicants are digitally available, and
can be checked for inclusion of the suspension notice.
It was found that letters remain a human one-time
activity and not reflected in the log.
There were audit statements that could be seen as
principles for a group of other audit statements. For
example, “ST47: The activities that will be execu-
ted if the grant is approved, must be described in a
SMART manner”. ST47 is a principle for a group of
audit statements that will be executed if the grant
is approved” to be Specific, Measurable, Assignable,
Realistic and Time-related.
4.1.5 Results of the Audit with Process Mining
In total, 27 statements out of 52 were made suitable
for process mining. This shows the degree of audit
support that can be achieved with process mining. In
case of the design of logs of processes for automated
audit, the degree of support can be increased.
Results of the audit with process mining are the
following. 23 of 27 filters of all verifiable audit state-
ments did have one or more non-compliant cases that
were included in the event log. In the Disco-tool, the
A Practical Extension of Frameworks for Auditing with Process Mining
411
Normative Process Model
Grant Application Process
Management Advisor Administrative
1. Open
Case File
X
3. Request
missing
information
2. Send
Confirmation
Letter
Start Process-
Application
received
5. Request
additional
information
6. Prepare
Decision
X
4. Review
Application
Complete?
yes
no
Clarification
needed?
7.Conform
Decision
8. Send
Acceptance/
Rejection letter
X
9. Send
request for
justification
Justification needed?
10. Sanction
11. Evaluate
Justification
X
On time?
12. Definitive
grant
allocation
yes
no
no
yes
yes
no
Figure 3: Initial Normative process.
Figure 4: Corrected grant application process.
non-compliant cases can be visually identified by ap-
plying the appropriate filter for the audit statement.
For example, after application of the ST 7 filter
(about deadlines for receiving missing information),
the process map, containing four cases, can be analy-
zed to find the reasons of non-compliance (Wiersma,
2017).
ST 7 has four cases that are non-compliant. Is
this good or bad? This depends on the risks associ-
ated with compliance to this statement. The analysis
should be done by the Audit and Business Experts.
4.1.6 Results of the Case Study
The audit of the Grant Application Process has shown
evidence that a participatory workshop is the key step
to the success of frameworks for audit with process
mining.
In the participatory workshop, the business model
described by business experts was corrected by con-
fronting with the activities recorded in the log. Audit
statements were rewritten using the names of activi-
ties in the log, and the norms used in the business
process.
After such preparations, the process mining was
used for compliance checks of the business process to
the audit statements.
4.2 Audit of a Process of Handling
Invoices
The second case study was conducted in a finan-
cial department, because the processes in this dom-
ain are a subject to law regulations and have guidance
proscribed by the national professional accountants
organizations, like, for example, (PAiE, 2017).
The question was if the identified need for a parti-
cipatory workshop depends on the domain of audit.
ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering
412
4.2.1 Evidence of the Need of a Participatory
Workshop
An organization uses services and purchases goods.
The service and goods providers send invoices as re-
quests for payment for certain goods or services. The
handling of purchasing invoices is a financial process.
We have found the following sources with some
elements of the process model: (a) a process break-
down in a table of statements that is used by the inter-
nal auditors; (b) a user process instruction for specific
roles.
We have collected the initial information about
process activities, their relation to business roles and
logs in a table (see Figure 5) to present it during the
participatory workshop. Figure 5 shows the absence
of a well-described business process model of the au-
dited process. The found audit statements were in the
form of control objectives. A participatory workshop
was needed.
4.2.2 Results of a Participatory Workshop
One result of the participatory workshop was a Nor-
mative Process. The limited space for the paper does
not allow us to present it. The process can be found
in (Wiersma, 2017).
Another result is the 55 audit state-
ments (Wiersma, 2017). Following our framework
extension, the statements were rewritten for compli-
ance checks using a Controlled Natural Language.
We faced with the fact that not all 55 statements
were unambiguous. For instance, statement ST5.1 is
ambiguous:
ST5.1. Imported invoices are rerouted to employees
that can select the department for further processing
of the invoice after the necessary information is ad-
ded.
Our sources contained no further information that
could be used to assert which data is considered ne-
cessary.
The participatory workshop has shown that no log
of the Purchasing Invoices Process is available. The
Purchasing Invoices Process workflow is implemen-
ted in the OpenText eDocs Document Management
System, which is used by all departments. In addition
to this system, the scanning and processing of recei-
ved invoices was done in a KTM (Kofax Transfor-
mation Module) application. Finally, the actual pay-
ment order to the bank is processed by the financial
system. Logging of this system was available for au-
diting, but was too complex to integrate in the audit.
To obtain the event log of the invoices workflow, IBM
Cognos BI was used to select and export the data from
the eDocs Document Management System. For this
case, a selection is made based on the time stamps
of case activities in eDocs with date values between
01 01 2015 and 31 12 2015.
Even in the found log, not all audit criteria could
be transformed into a filter of a process mining tool,
because the log did not contain necessary informa-
tion. The first two activities of the process (scan-
ning and importing) are not part of the activities in
the event log as these are performed in a different sy-
stem. As a result, we could use only 11 out of 55
audit statements for the process mining and finding
the compliant and non-compliant cases.
5 RESULTS AND CONCLUSIONS
In this work, we have evaluated the upcoming techno-
logy of audit with process mining.
Analysing the literature, we have found frame-
works for audit with process mining that assume exis-
tence of business experts with the knowledge of a bu-
siness process model, and compliance experts, who
are able to formulate audit statements applicable for
process mining tools.
Conducting the case studies, we have found that
the companies more likely have IT Experts, Audit
Experts and Business Experts. The success of ap-
plication of process mining for audit depends on
agreement on terminology achieved by these three
groups of experts. We assumed that this agreement
can be achieved in a participatory workshop.
We have proposed to include such a workshop in
any practical audit framework with process mining in
order to manage the expectations of businesses, that
plan to apply process mining for audit.
We have applied an extended framework in two
case studies in different business domains and succee-
ded to design the normative business processes and
audit statements that can be used for compliance
checks on logs.
Both case studies have evidenced that the direct
application of process mining for audit is a too large
step that cannot be made by the business without in-
termediate steps of designing the normative business
processes and audit statements applicable for process
mining on the process logs. In our first case study, the
business process model provided by business experts
was corrected by extra outputs of decisions. The cor-
rection came from the log analysis. In the second case
study, the business process model was created from a
table of partial descriptions of process steps.
Both of our case studies have evidenced that such
a business role as ”Compliance Expert” does not exist
and the initial audit statements made by Audit Experts
A Practical Extension of Frameworks for Auditing with Process Mining
413
Nr
Name
Description
Role
Activity in log
1.
Scan invoice
Incoming invoice (by mail) is scanned as part
of a batch of invoices
Department
Invoices
None (Kofax
system)
2.
Check / complete
scanned invoice
The text on the invoice is automatically
recognized (OCR) and translated to specific
fields: date on the invoice, invoice no., bank
account number, commitment number and
amount payable. The result of the OCR is
visually checked against the digital scan (and
completed / corrected)
Department
Invoices
None (KTM
system)
3.
Send invoice to
DMS
The invoice is exported from KTM and
imported into the Document Management
System (eDocs)
Department
Invoices
Purchasing
Invoices 2.0
4.
Complete invoice
details
The fields required for further handling that
were not imported are completed: invoice
description, tax code and amount,
department to authorise, number of supplier
(looked up in Decade)
Financial data
entry
Purchasing
Invoices 2.0
5.
Invoice at block
manager
The invoice is sent to the block manager for
re-routing to the department
Financial data
entry
1.Invoice at block
manager
6.
Re-route to
department
The block manager re-routes the invoice to
the department for authorization
Block manager
2.Receive and
reroute
7.
Encode invoice
The invoice is re-routed to the right resource
for the performance check, to verify that the
goods that are billed are delivered.
Department
inbox
3.Encode invoice
X
Correctly Booked?
If the department thinks the invoice is
incorrectly put in their inbox, it is rerouted
back to the block manager.
8.
Performance
check
The performance check is executed by
sending an ‘Ok’ and a comment in the eDocs
system. This step is not enforced and
sometimes skipped.
Checker
4.Performance
check
9
Encode invoice
The booking details (cost centre and booking
combination) are entered.
Department
inbox
3. Encode invoice
10.
Check booking
details
The invoice is sent to a resource that checks
the booking details. This step can be skipped
Checker
5. Booking entry
check
11.
Approval
The invoice is approved by the budget
administrator (in case of a commitment
already exists in Decade) or the budget
manager. Approval is done for each invoice
line.
Budget admin /
budget mgr.
6. Approve invoice
12
Authorize payment
/ process invoice
The complete invoice and payment details
are checked and payment is authorised. After
authorisation, the payment is automatically
processed by Decade.
Financial
advisor
Process invoice
Figure 5: Table for the Participatory workshop.
should be rewritten for process mining by an IT expert
in collaboration with an Audit Expert and a Business
Expert.
So, the cases demonstrate usefulness of our pro-
posal to include a participatory workshop into audit
frameworks for audit with process mining. They also
show the types of artifacts that are created by the par-
ticipatory workshop: a Normative Business Process
Model using the activities mentioned in the log and
a set of Audit Statements formulated in a Controlled
Natural Language. A Controlled Natural Language
uses the activities mentioned in the log and the com-
pliance patterns that can be checked in the log by pro-
cess mining tools.
The construct validity of our case studies is gua-
ranteed by the many sources of information used for
conduction of them (triangulation). Indeed, any busi-
ness model and statement was inspected by three dif-
ferent specialists using different documents.
The conduct validity of our case studies can be
checked by everyone as we have presented the steps
of the proposed framework and results for research re-
production, as it is recommenced in literature on case
studies (Yin, 1994). All details about case studies can
be found in (Wiersma, 2017). The logs can be provi-
ded on request.
ENASE 2018 - 13th International Conference on Evaluation of Novel Approaches to Software Engineering
414
Although we used the Disco tool for process mi-
ning, our extension is tool independent and can be
applied for the frameworks that work with different
process mining tools.
As future work, we invite using our extension for
audit frameworks with process mining. This may re-
sult in a reusable collection of normative business
processes and audit statements for specific domains
of audit application.
REFERENCES
Accorsi, R. and Stocker, T. (2012). On the exploitation of
process mining for security audits: the conformance
checking case. In Proceedings of the 27th Annual
ACM Symposium on Applied Computing, pages 1709–
1716. ACM.
Awad, A., Weidlich, M., and Weske, M. (2009). Speci-
fication, verification and explanation of violation for
data aware compliance rules. Service-Oriented Com-
puting, pages 500–515.
Barnawi, A., Awad, A., Elgammal, A., Elshawi, R., Alma-
laise, A., and Sakr, S. (2016). An anti-pattern-based
runtime business process compliance monitoring fra-
mework. framework, 7(2).
bupaR (2017). Business Process Analysis in R.
http://bupar.net/.
DISCO (2016). Flexicon DISCO.
http://fluxicon.com/disco/.
Jans, M., Alles, M., and Vasarhelyi, M. (2013). The case
for process mining in auditing: Sources of value ad-
ded and areas of application. International Journal of
Accounting Information Systems, 14(1):1–20.
Jans, M., Alles, M. G., and Vasarhelyi, M. A. (2014). A
field study on the use of process mining of event logs
as an analytical procedure in auditing. The Accounting
Review, 89(5):1751–1773.
Karapetrovic, S. and Willborn, W. (2000). Generic audit
of management systems: fundamentals. Managerial
Auditing Journal, 15(6):279–294.
Ly, L. T., Maggi, F. M., Montali, M., Rinderle-Ma, S., and
van der Aalst, W. M. (2013). A framework for the
systematic comparison and evaluation of compliance
monitoring approaches. In Enterprise Distributed Ob-
ject Computing Conference (EDOC), 2013 17th IEEE
International, pages 7–16. IEEE.
Mintzberg, H. and McHugh, A. (1985). Strategy formation
in an adhocracy. Administrative science quarterly, pa-
ges 160–197.
PAiE (2017). Professional Accountants in Europe.
http://paie.nl.
ProM (2016). Process mining Workbench ProM.
http://www.promtools.org.
Roubtsova, E. E. (2005). Property driven mining in work-
flow logs. In Intelligent Information Processing and
Web Mining, pages 471–475. Springer.
Russell, J. (2006). Process auditing and techniques. Quality
Progress, 39(6):71–74.
Sadiq, S., Governatori, G., and Namiri, K. (2007). Mo-
deling control objectives for business process compli-
ance. Business process management, pages 149–164.
Sandkuhl, K., Stirna, J., Persson, A., and Wißotzki, M.
(2014). Enterprise modeling. Tackling Business Chal-
lenges with the 4EM Method. Springer, 309.
Spreeuwenberg, S. and Healy, K. A. (2009). Sbvrs ap-
proach to controlled natural language. In Internatio-
nal Workshop on Controlled Natural Language, pages
155–169. Springer.
van der Aalst, W., van Hee, K. M., van der Werf, J. M., and
Verdonk, M. (2010). Auditing 2.0: Using process mi-
ning to support tomorrow’s auditor. Computer, 43(3).
van der Aalst, W. M. P. (2011). Process Mining - Discovery,
Conformance and Enhancement of Business Proces-
ses. Springer.
van der Aalst, W. M. P. (2012). Process mining: Overview
and opportunities. ACM Transactions on Management
Information Systems (TMIS), 3(2):7.
Wiersma, N. (2017). The use of Process Mining in Business
Process Auditing, http://hdl.handle.net/1820/7702.
Open University of the Netherlands.
Yin, R. (1994). Case study research: Design and methods .
beverly hills.
A Practical Extension of Frameworks for Auditing with Process Mining
415