KPIs 4 Workplace Learning
Sandro Emmenegger, Knut Hinkelmann, Barbara Thönssen and Frieder Witschel
University of Applied Sciences and Arts Northwestern Switzerland, School of Business, Olten, Switzerland
KEywords: Model Driven Engineering, Workplace Learning, Learning Scorecard, Ontology, KPI.
Abstract: Enterprises and Public Administrations alike need to ensure that newly hired employees are able to learn the
ropes fast. Employers also need to support continuous workplace learning. Workplace learning should be
strongly related to business goals and thus, learning goals should directly add to business goals. To measure
achievement of both learning and business goals we propose augmented Key Performance Indicators (KPI).
In our research we applied model driven engineering. Hence we developed a model for a Learning
Scorecard comprising of business and learning goals and their KPIs represented in an ontology. KPI
performance values and scores are calculated with formal rules based on the SPARQL Inferencing Notation.
Results are presented in a dashboard on an individual level as well as on a team/group level. Requirements,
goals and KPIs as well as performance measurement were defined in close cooperation with Marche
Region, business partner in Learn PAd.
1 INTRODUCTION
Within the European funded project Learn PAd a
model-based approach was developed that supports
collaborative workplace learning. Workplace
learning is considered strongly related to workplace
performance, i.e. all learning should contribute to
improve work results. In our research we investigate
how learning can be (1) related to business goals, (2)
how workplace learning can be measured and (3)
how such an approach can be automated. In our
approach, we determine learning goals that support
the achievement of business goals and derive Key
Performance Indicators (KPIs) to measure
achievement of business and learning goals. Next, it
is determined who – i.e. which organisational units
and business roles – is supposed to meet the goals
and which competencies are required from the roles.
This approach allows for deriving general
learning goals for an employee, as well as personal
learning goals derived from the gap between
acquired competencies and the required
competencies of a role. It also allows to assess an
employee’s workplace learning progress based on
the measurement of the KPIs.
2 RELATED WORK
As shown by Wang et al. (2010) in many
organizations, e-learning is not aligned with the
organizational vision and mission. Focus is put on
technical aspects neglecting motivation and
assessment of the learners. The authors elaborate on
embedding learning activities in the workplace to
address corporate interests (organization), individual
needs (learner), work performance (work), and
social context (other learner).
Nikolova et al. (2014) did a comprehensive
literature review and showed that most research
done on measuring workplace learning is limited by
its context dependence. According to Nikolova et al.
(2014) workplace learning has two main
components: an interactional and a task-based one.
However, contrary to the approach pursued in Learn
PAd, task-based is used in the notion of cognitive–
behavioural but not in the sense of getting better in
performing a (business process) activity. Hence,
learning goals and measures remain unrelated to
business goals.
In research done by van Dam (2015) workplace
goal orientation is investigated, distinguishing
between learning, performance and avoidance. That
is, workplaces emphasizing learning goals are likely
to provide opportunities for personal growth, like
challenging job assignments and learning activities;
Emmenegger, S., Hinkelmann, K., Thönssen, B. and Witschel, F.
KPIs 4 Workplace Learning.
DOI: 10.5220/0006090902630270
In Proceedings of the 8th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K 2016) - Volume 3: KMIS, pages 263-270
ISBN: 978-989-758-203-5
Copyright
c
 2016 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
263

workplaces emphasizing performance goals are
likely to impose pressure on employees and show a
high degree of comparison and competition;
workplaces emphasizing avoidance goals are likely
to focus on punishing errors (van Dam 2015). Also
in this research goal orientation is not considered
with respect to supporting a learner in better
reaching an organisation’s business goals.
Workplace learning in a broader context of an
organization like the political economy in which
goods or services are sold, economic sectors and
structure of production was researched by Fuller and
Unwin (2011). Although Fuller and Unwin (2011)
provide a comprehensive framework for capturing
organisational factors which influence how people
learn at work and how this learning can be valued,
fostered or limited, they spare the ‘measurement
challenge’ (quotation marks by the authors).
2.1 Learner Assessment Strategies
In their approach Faddouli et al. (2011) enhanced
previous work on formative assessment which
allows for personalized learning. Assessment is done
based on offered items (i.e. questions) presented to
the learner. For each assessed item the competency
gap is identified, i.e. the gap between current level
of performance and target level of performance in
order to identify a suitable next learning activity.
Faddouli et al. (2011) differentiate between static
level (captured in a profile) and dynamic level of a
learner (describing the learning progression). Within
the Learn PAd project a similar approach is pursued:
the (more) static level is also captured in a learner’s
profile whereas the dynamic level is represented in
the Learning Scorecard. As we regard learning as a
collaborative process, assessment of individuals is
not enough: a learner’s performance must be
assessed within the context of a (learning) team
performance. Hence, in our approach we exceed the
outcome of Faddouli et al. (2011) as not only
learning performance of individuals but also from
team/groups, i.e. organizational units is considered.
The purpose of assessment for learning is “to
monitor the progress of the learner toward a desired
goal, seeking to close the gap between a learner’s
current status and the desired outcome” (Clark 2012,
p 208). In his comprehensive contribution Clark
(2012, p 208) also shows that assessment can be
regarded as learning: A process in which learner
and teacher “set learning goals, share learning
intentions and success criteria, and evaluate their
learning through dialogue and self and peer
assessment” (2012, p 208). In the Learn PAd project
this notion is transferred into workplace learning,
supposing that learning goals are 1) aligned with
business goals and 2) measured via KPIs related to
those business goals which in turn support the
strategic goals of an organization.
Wang et al. (2011) suggest to consider the
alignment of individual and organizational learning
needs, the connection between learning and work
performance, and communication among individuals
when designing workplace e-learning. They set up a
set of key performance indicators (KPIs) with
measures “focusing on the aspects of organizational
and individual performance that are critical for the
success of the organization” (2011, p 167).
2.2 Knowledge Maturing Scorecard
Within the MATURE project a Knowledge
Maturing Scorecard was developed (Hrgovcic &
Wilke 2012). Knowledge maturing (Schmidt et al.
2012) describes a process of learning on a collective
level, which consists of various phases, where
knowledge reaches ever higher degrees of
sophistication and organisational acceptance. The
Knowledge Maturing Scorecard follows the
principles of a Balanced Scorecard (Kaplan &
Norton 1996), but replaces strategic goals with
knowledge maturing goals and key performance
indicators with knowledge maturing indicators.
Although the approach of using a (modified)
Balanced Scorecard may be adequate to measure
knowledge maturing, it does not model learning
goals and their relations to business goals and hence
does not allow for assessing learning with respect to
improving business performance.
3 RESEARCH METHODOLOGY
For our work we followed the design science
research methodology for information systems
research (Hevner et al. 2004). Hence, the research
design follows the following stages:
In the ‘Awareness of Problem’ phase we
performed a detailed domain analysis to understand
which goals and KPIs are relevant for measuring
learning performance in a workplace environment.
In the ‘Suggestion’ phase we derived and described
the conceptual models that facilitate the
implementation of goal oriented learning at the
workplace.
In the ‘Development’ phase we defined and
implemented the technical architecture for learning
performance monitoring. All artefacts were
KMIS 2016 - 8th International Conference on Knowledge Management and Information Sharing
264

iteratively developed in close cooperation with the
business partner, a Public Administration, in the
Learn PAd project.
The solution will be fully evaluated (‘Evaluation’
phase) within the upcoming final phase of the Learn
PAd project.
4 ASSESSING WORKPLACE
LEARNING
As our work is part of the Learn PAd project,
workplace learning is investigated for the
application domain of Public Administrations (PAs).
PAs must perform complex processes in order to
provide services to citizens and companies.
Complexity stems from several issues: e.g. new or
updated laws and regulations require creation or
adaptation of services and processes and many
activities must be performed collaboratively by
different, possibly many, PA offices. To come to
grips with his/her assignment is tedious for a
beginner and public administrators are never done
with learning how to carry out their tasks.
In the following we will focus on how learning
goals are determined and learning progress is
measured.
4.1 Goals and KPIs
We followed a top-down approach starting from
strategic business goals, which are supported by
operational business goals, which are supported by
learning goals. For operational and learning goals
we then identified the KPIs and how to measure
them. Since in Learn PAd we pursue a model-driven
approach we consider three model kinds relevant for
our approach: the Business Motivation Model
(BMM) (OMG 2014), the Learning Scorecard
Model and the Organisational Model. In the
Learning Scorecard Model operational business
goals and learning goals and their KPIs are
modelled. Business goals are related to one or more
motivation element(s) of BMM (e.g. (strategic) goal,
objective, and target). Furthermore, for each
organisational unit and role, the operational und
learning goals to be achieved are determined.
4.1.1 Learning Scorecard
Like the Balanced Scorecard (Kaplan & Norton
1996), our Learning Scorecard considers four
perspectives: Client, Process, Financial and Learning
Organisation. Perspectives may contain business
goals and their relation to strategic goals (modelled
in the BMM) and a new type of goal – the learning
goal, which supports one or more business goals.
We created an organisation-specific model
together with business representatives of a PA and
extended the properties of KPIs to be able to model
assessment of learning. Figure 1 depicts a part of the
Learning Scorecard showing business and learning
goals and their KPIs for the Client and Process
Perspective.
Figure 1: Parts of Learning Scorecard.
KPIs 4 Workplace Learning
265

Operational business goals are represented by
orange circles (their relation to strategic goals is not
visible in the figure); arrows indicate how one goal
may support another one. Learning goals supporting
the business goals are represented by striped
triangles, for example ‘Acting Responsibly’ supports
‘High quality of services’. In addition a learning
goal can also support another learning goal as
depicted in the lower part of the figure. KPIs are
represented by targets, e.g. ‘no of complaints of
clients’ is a KPI for the business goal ‘High quality
of service’. The red rectangles in Figure 2 indicate
the goals and KPIs detailed in Table 1.
It was also differentiated between KPIs for
individuals (which can be aggregated on team level)
and KPIs specific for teams (organisational units,
which again can be aggregated on department level
and so forth). Therefore we assign business and
learning goals to roles and organisational units. For
our chosen scenario, 7 business goals measured by
27 KPIs and 12 learning goals measured by 18 KPIs
were determined overall.
4.1.2 Measurement of KPIs
To measure KPIs we consider three types of sources:
external data, user activity log, and simulation.
External Data: for several KPIs, relevant
information resides outside of the reach of the Learn
PAd system (e.g. stored in PA’s legacy systems). In
many cases, the information might not be readily
available in electronic form at all, e.g. because it
partially depends on subjective assessment of a
human (e.g. KPI ‘Acting autonomously on one's
own responsibility’). In such cases, we assume that
learners will discuss the assessment of the KPI e.g.
as part of regular performance reviews and that the
value will then be stored in a commonly used
spreadsheet.
User Activity Log: many KPIs refer to the way
the Learn PAd system is used for workplace
learning. In particular, these KPIs assess whether
learners extend and contribute their knowledge by
using functionalities of the system and whether they
contribute to process improvements through
feedbacks.
Simulation: Some KPIs assess to what degree
learners reach learning goals in simulations within
the Learn PAd simulation environment.
The calculated KPI values of individual learners
as well as the figures on organisational levels are
shown in a dashboard. In Figure 4 an example of the
learner's individual dashboard with calculated KPI
scores is shown.
The user can drill down from the aggregated
levels, like the perspectives, to the leaves, the KPIs.
On the left hand side of the figure the entities of the
Learning Scorecard are provided in a hierarchical
structure, starting from the perspectives followed by
the business goals, the learning goals und their KPIs.
Performance is depicted in form of lights, followed
by information about trend, unit, target and current
value. In case of severe underperforming (red light)
recommendations for improvement are provided.
With this approach we provide a set of meta
models that allow for explicitly defining learning
goals and their relations to business goals and use
well-established methods (aka KPIs) for assessing
workplace learning. Details of the KPI calculations
and implementation of the dashboard are provided in
the following sections.
Table 1: Examples of Goals, KPIs and their Attributes.
Goal
KPI
Measurement
Lights / Threshold
Unit
Period
Business goal:
High quality of
services
no of
complaints of
clients (about
an employee /
learner)
Self-assessment:
interpretation of customer
feedback
Green: <=20%
%
30 days
Orange: >20% <=40%
Red: >40%
Learning goal:
Familiarity with
Learn PAd
functionality
global action
per user
Log: number of
interactions with Learn
PAd platform in 30 days
(i.e. no of comments + no
of additional pages + no of
pages navigated)
Green: >=12
#
3 months
Orange: >=5 <12
Red: <5
business
process
simulation
score
Simulation: ratio of
achieved business process
score to the maximum of
business process score
Green: >=70%
%
30 days
Orange: >=50% <70%
Red: <50%
KMIS 2016 - 8th International Conference on Knowledge Management and Information Sharing
266

Figure 2: KPI model and instance layers.
4.2 Ontological Representation
Figure 2 shows how models and instances of the
learning scorecard are being represented in the web
ontology language OWL (OWL Working Group
2012): KPI concepts are modelled in the meta model
layer M2. Examples of domain specific instances of
KPIs are depicted in layer M1.
The KPI value and score calculation during
system runtime is based on gathered runtime data,
the background knowledge provided by the domain
ontology and on data from other integrated systems
– shown as instances of KPIs in layer M0.
Here, we face the problem of a missing support
of multilevel modelling by the ontology description
standards, like OWL. We have an instance of an
instance problem if we add KPI value instances with
calculated scores to our ontology, where the KPIs
and other model instances are in turn instances of the
meta model concepts of the highest layer. Following
Fanesi (2015) and Fanesi et al. (2015) who have
shown an approach how to overcome that problem
and still keep it decidable by reasoners, the KPI
model instances are modelled as instances and
classes at the same time.
KPI values are calculated on an individual level,
i.e. for each employee, and on an org unit level.
4.3 Implementation (EMS)
The concrete KPIs are modelled by knowledge
engineers in an extended standard modelling
environment. For this project the Meta Modelling
Platform AdoXX is used. For further analysis and
calculations, the exported models from the
modelling environment are transformed with XSLT
into instances of the ontology (step 2 in Figure 3).
The KPI calculation is based on data gathered during
runtime of the Learn PAd system (see Section 4.1.2).
KPIs 4 Workplace Learning
267

Figure 3: KPI system components.
The KPI performance values and scores are
calculated with formal rules based on the SPARQL
Inferencing Notation (SPIN) (W3C n.d.). SPARQL
(Prud’hommeaux & Seaborne 2008) is the query
language for RDF based models, and therefore for
our ontology, and has been standardized by the
World Wide Web Consortium (W3C). The
inferencing engine used is provided as open source
implementation by TopBraid (TopQuadrant n.d.)
Several rules are applied to calculate and assert
the KPI performance value instances with the
property's value, score, trend, timestamp and
assigned business actor. The rules are executed
iteratively (depending on their level) and
incrementally (run as long as new instances can be
inferred). This means that the rules consider inferred
values of the previous iteration. Another advantage
of SPIN rules is their representation in the RDF
format which allows assigning and storing them
directly in the ontology.
Level #1 Rules
The first level rules consider collected runtime data
stored in the ontology repository. For instance the
KPI "Global actions per user" considers all activities
on a user on the Learn PAd platform, like the
feedbacks for improvements, comments,
attachments etc. a user has provided during the
usage of the learning platform. The rule applied for
this KPI counts simply all logged activities of a user.
The values from the external data sources provide
directly a KPI value and do not have to be
calculated.
Level #2 Rule
In a second step the KPI value score is calculated
based on the previously calculated and inferred
actual KPI values and the loaded KPI values. The
score can be 1, 2 or 3 and represents a traffic light in
the dashboard (1=red, 2=yellow, 3=green). The rule
considers the thresholds defined in the modelled
KPIs and is a generic rule applied for all KPI
performance value instances:
Level #3 Rule
This rule infers the performance properties on a
higher aggregation level, like the organisational
units. The minimum function is applied, means if at
least one KPI of a sub unit or an employee assigned
to the units is red, then the organisational units KPI
score goes also on red.
All the inferred values will finally be exported to the
dashboard files. The relationship between learning
goals, learning material and calculated KPI
performance scores for individuals enables new
recommendations to be provided. On the one hand,
KMIS 2016 - 8th International Conference on Knowledge Management and Information Sharing
268

Figure 4: KPI dashboard with learning recommendation.
this includes recommendations for individual
learners, suggesting learning material or activities to
improve a bad performance score (red traffic light)
of a KPI. An example of the dashboard with such a
recommendation is shown in Figure 4.
4.4 Application Example
Let us suppose that a PA in a region in Italy hires a
new employee – Gianni – as a SUAP officer. With
his boss Sarah, he determines the business goals
assigned to and the competencies required for this
role. The gap between the required competencies of
the role and Gianni’s actually acquired competencies
determines his individual learning goals. Gianni and
Sarah review Gianni’s learning goals and
corresponding KPIs and define how they could be
met in what timeframe. Sarah also explains the goals
of the organisational unit to show how Gianni’s
performance contributes to the team performance. In
the following Gianni’s performance is monitored
and he can consult the dashboard (see Figure 4) at
any time to check on his improvements. His boss
Sarah can do the same for the whole team as well as
for the individual members of her team.
4.5 Pre-evaluation
Since the KPI related models were developed in the
last phase of the Learn PAd project which ends in
November 2016, full evaluation will be performed
within the next weeks with at least 30 PA officers.
So far, evaluation has been done for intermediary
results. Thus, all artefacts were cooperatively
developed and constantly assessed by the business
partner in the Learn PAd project.
Therefore we conducted interviews and
workshops with the business representatives and
they confirmed the utility and correctness of every
artefact (requirements, design and implementation of
goals, KPIs and relations amongst entities as well as
sources for, content of and representation of the
dashboard).
5 CONCLUSIONS
Our approach for goal-oriented workplace learning
based on KPIs provides a series of conceptual and
technical advances, designed to support
organisations in planning and tracking the learning
progress of employees. With our approach, we are
able to explicitly relate workplace learning to
business goals, to measure learning with regards to
meeting business goals and to automate assessment
and display of learning progress. This includes the
design of a new meta-model for learning scorecards
– allowing organisations to connect learning goals to
organisational goals and to model certain special
aspects of learning KPIs, including e.g. learning
recommendations to help learners when they fail to
reach KPI target values. Furthermore, an exemplary
and at the same time generic learning scorecard has
KPIs 4 Workplace Learning
269

been derived from a domain analysis across public
administrations. Finally, several methods for the
assessment of learning outcomes and goal
achievement have been described. A prototypical
implementation has been performed and
corresponding technical details have been described.
The prototype implementation is made available on
the Learn PAd github project including the ontology
files that build together the Learn PAd domain
ontology and cover the enterprise upper ontology
files, the Learn PAd specific domain files and the
KPI models with rules.
ACKNOWLEDGEMENTS
This research was supported by the EU through the
Model-Based Social Learning for Public
Administrations (Learn Pad) FP7 project (619583).
REFERENCES
Clark, I., 2012. Formative Assessment: Assessment Is for
Self-regulated Learning. Educational Psychology
Review, 24(2), pp.205–249.
Van Dam, K., 2015. Workplace Goal Orientation.
European Journal of Psychological Assessment, 31(1),
pp.62–68.
El Faddouli, N. et al., 2011. Towards an adaptive
competency-based learning system using assessment.
International journal of computer science, 8(1),
pp.265–274.
Fanesi, D., 2015. A Multilayer Ontology to represent
Business Process Models and Execution Data.
Fanesi, D., Cacciagrano, D.R. & Hinkelmann, K., 2015.
Semantic Business Process Representation to enhance
the degree of BPM mechanization an ontology. In K.
Hinkelmann & B. Thönssen, eds. Third International
Conference on Enterprise Systems ES2015. IEEE
Computer Society Publications.
Fuller, A. & Unwin, L., 2011. Workplace Learning and
the Organisation. In M. Malloch et al., eds. The SAGE
handbook of workplace learning. pp. 46–73.
Hevner, A.R. et al., 2004. Design Science in Information
Systems Research. MIS Quarterly, 28(1), pp.75–105.
Hrgovcic, V. & Wilke, G., 2012. The Knowledge
Maturing Scorecard: A model based approach for
managing the innovation. In Computer Science and
Information Systems (FedCSIS), 2012 Federated
Conference on. pp. 1141–1148.
Jia, H. et al., 2011. Design of a performance-oriented
workplace e-learning system using ontology. Expert
Systems with Applications, 38(4), pp.3372–3382.
Kaplan, R.S. & Norton, D.P., 1996. The balanced
scorecard: translating strategy into action, Harvard
Business School Press.
Nikolova, I. et al., 2014. Work-based learning:
Development and validation of a scale measuring the
learning potential of the workplace (LPW). Journal of
Vocational Behavior, 84(1), pp.1–10.
OMG, 2014. Business Motivation Model. BMM 1.2,
(May), p./.
OWL Working Group, 2012. Web Ontology Language
(OWL).
Prud’hommeaux, E. & Seaborne, A., 2008. SPARQL
Query Language for RDF.
Schmidt, A. et al., 2012. Knowledge Maturing: Creating
Learning Rich Workplaces for Agile Organizations A.
Schmidt & C. Kunzmann, eds., Available at:
http://knowledge-maturing.com/files/whitepaper.pdf.
TopQuadrant, TopBraid Composer.
W3C, SPIN SPARQL Inferencing Notation.
Wang, M. et al., 2010. A performance-oriented approach
to e-learning in the workplace. Educational
Technology and Society, 13, pp.167–179.
KMIS 2016 - 8th International Conference on Knowledge Management and Information Sharing
270