A FRAMEWORK TO MEASURE THE IMPACT OF

KNOWLEDGE PROCESS SUPPORTING TECHNOLOGIES

An Integrated Approach for a Technology-oriented Business Benefit Analysis

Marcel Scheibmayer, Ali Imtiaz and Martin Bleider

Research Institute for Operations Management (FIR), RWTH Aachen, Pontdriesch 14/16, Aachen, Germany

Keywords: Knowledge management, Information sharing, Business benefit, Business model, Knowledge processes.

Abstract: Within the last decade a plethora of different technologies emerged claiming to support knowledge worker

in their everyday working life. Based on these technologies many different solutions for knowledge process

support in enterprise environments have been developed. However the impact of these solutions is mostly

considered intangible due to the fact that the overall costs and benefits are not clearly traceable. Traditional

approaches for the evaluation of the costs and benefits often cover only a part of the overall scope, leading

to an imminent need for a fitting assessment methodology. This paper presents a research approach to

develop an integrated cost-benefit-framework, which integrates traditional aspects and related

methodologies with emerging, knowledge process based aspects.

1 INTRODUCTION

Due to increasing competition in global markets,

enterprises are constantly facing the need not only to

revisit their products and services, but also their

processes. While great enhancements in productivity

have been achieved by formalizing business

processes (Hill, Yates, Jones, Kogan, 2006),

Knowledge experts, who primarily deal with

informal knowledge processes, are weakly supported

on the enterprise level (Brown, 2007). They often

depend on tacit knowledge and undocumented,

unformalized knowledge processes. To enhance the

productivity of knowledge worker it is essential to

make important information easily accessible.

Therefore, the fields of Knowledge Management and

Information sharing are constantly gaining

importance for knowledge process support.

From a system perspective, users often had to

rely on personal communication and ad hoc

collaboration to receive the information needed.

Those situations were leading to context switching

and a loss of time and productivity for involved

people. With the appearance of web 2.0, a plethora

of different technologies arose that offer new

possibilities to support knowledge processes. Instant

messengers, wikis, weblogs, social networking tools

and podcasts are common tools which are also

available for knowledge workers in enterprise

environments. However recent reports show that an

extension of numbers of tools does not necessarily

enhance the productivity (Forrester Big Idea, 2006).

Current research approaches focus on simpler,

yet richer integrated workspace solutions on the one

hand and formalizing knowledge on the other hand

(Active, 2008). New developments in the fields of

collaborative technologies, context aware

technologies and ontologies enable new possibilities

of knowledge process support.

Prior to invest into these arising technologies or

during the evaluation of the existing IT support,

organisations need to assess the impact on their own

processes and evaluate the related cost and benefits

coming along. However existing tools and

methodologies often cover only a small part of the

relevant aspects, especially regarding knowledge

process related aspects coming along with these

emerging technologies and concepts. This fact is

quite problematic and companies are often left with

an estimated guess as a basis for decision.

This paper embraces a new approach to assess

the impact of knowledge process supporting

technologies, focusing on knowledge process related

aspects that are hardly covered by existing

methodologies.

253

Scheibmayer M., Imtiaz A. and Bleider M..

A FRAMEWORK TO MEASURE THE IMPACT OF KNOWLEDGE PROCESS SUPPORTING TECHNOLOGIES - An Integrated Approach for a

Technology-oriented Business Beneﬁt Analysis.

DOI: 10.5220/0003116702530259

In Proceedings of the International Conference on Knowledge Management and Information Sharing (KMIS-2010), pages 253-259

ISBN: 978-989-8425-30-0

 2010 SCITEPRESS (Science and Technology Publications, Lda.)

2 STATE OF THE ART

& RESEARCH PROBLEM

Determining the impact of technologies on

knowledge process support in form of costs and

benefits is a highly complex task that includes

various aspects from different areas of expertise.

According to Ramirez and Nembhard there are no

effective methodologies to measure the productivity

of knowledge worker (Ramirez, Nembhard, 2004).

Thus leading to the situation that organizations

dealing with knowledge processes either not

measure them (Ahmed et al., 1999) or use

established cost and benefit models, that have

proven themselves in the field of manufacturing, to

evaluate their investments in information and

communication technology (Pietsch, 2003). The

most frequently used traditional methodologies are

introduced in the following section.

One of the earlier methodologies for cost benefit

analysis is Return-on-Investment (ROI), which

analyses the net benefits and divides them by the

overall costs. Using this profitability equation ROI

can identify past performance or future expectations

(Schachner, 1973). The methodology Total Cost of

Ownership (TCO) has the aim to identify every cost

driver generated by using information and

communication technology during their complete

lifecycle and evaluate them in a financial manner. It

considers direct and indirect costs. While direct

costs are easily to asses, for the assessment of

indirect costs the Gartner Group recommends using

interviews and surveys (Pietsch, 2003). The Hedonic

Wage Model analyses shifts in each job profile

towards more value creating activities, thus

determining the benefit in a monetary manner

(Pietsch, 2003). As a more process-oriented model,

Activity Based Costing splits the direct costs into

several parts and allocates each part to the process

that determines the costs (Pietsch, 2003). In doing

so, processes costs become more transparent as their

sources can be traced back to the respective sources.

To determine the potential benefits and hence

evaluate investments in fields of knowledge

management, organization need more than the

traditional financial measurements. An exclusive

technological perspective leads to a neglect of the

true potential benefits of knowledge management

(Ahmed et al., 1999). According to Ahmed, Lim and

Zairi, characteristics of a good knowledge

measurement system are:

– performance is reflected at various levels of

organisational systems. It is measured at the

strategic, tactical and operational levels;

– performance measurement is a distributed

activity reflecting various levels of ownership

and control;

– performance measurement reflects a blend of

measures for individual tasks/activities to

manage processes;

– performance measurement highlights

opportunities for improvement in all areas with

leverage points.

Therefore a series of approaches have been

developed. Ahmed, Lim and Zairi developed a

matrix-system that considers a lot more than the

financial ratios. Their approach regards four

different perspectives on knowledge management,

the customer, organization, supplier and technology

perspective. Within these perspectives the fields

capture, share, measuring and learning shall be

measured by various key data, thus giving managers

a more transparent illustration of used knowledge

management systems than a traditional financial

approach (Ahmed et al., 1999).

The approach of Haas and Hansen differentiates

between two types of knowledge sharing: electronic

documents and personal advice. The knowledge

sharing dimensions are separated in content - level

of quality and process - level of rework for

electronic documents and content - level of

experience and process - lack of effort for personal

advice. As measurable benefits Haas and Hansen

identified time saved on task within the type of

electronic documents and quality of work and signal

of competence within the type of personal advice

(Haas, Hansen, 2007).

The three steps model of Eschenbach and

Schauer aims at identifying potentials for improved

knowledge work productivity. In step one the

knowledge intensity of the organisation is identified

by measuring information intensity,

interdependencies, variability, innovation rate and

qualification requirements. In step two the current

level of knowledge work productivity is detected by

interviews regarding information processing,

communication, decision making and the adaptation

of an organisation to changing circumstances. In

conclusion step three analyses fields for

improvement (e.g. knowledge exchange, double

loop learning) (Eschenbach, Schauer, 2008).

Even though there are specific knowledge

management measuring approaches, often they are

not able to quantify their conclusions and are not

able to explicitly implicate potential benefits of

evolving technologies in the fields of collaborative

technologies, context aware technologies and

KMIS 2010 - International Conference on Knowledge Management and Information Sharing

254

ontologies.

Altogether existing models alone cover only

fragments of the full scope of knowledge process

support. There is an imminent need for a fully

integrated methodology which goes beyond

traditional approaches and combines traditional cost

and benefit aspects with as yet neglected aspects

coming along with emerging technologies,

knowledge management and information sharing.

The related research questions are:

– How can the impact of emerging collaborative

technologies for knowledge process support be

measured?

– What are the relevant cost driver and benefits

that have to be considered and how can they be

assessed?

– Which existing methodologies can be integrated

in this approach?

The next section introduces the research

environment as well as the research approach on this

topic.

3 RESEARCH ENVIRONMENT

AND RESEARCH APPROACH

The ACTIVE project aims to support knowledge

worker by leveraging tacit and unshared knowledge,

the “hidden intelligence of enterprises”, and convert

it into transferable and actionable knowledge

(Active, 2008). Therefore, concepts and tools from

the fields of social software and Web 2.0, Semantic

Technologies, Context Mining, Context Modelling,

and Context Sensitive Task Management, as well as

Knowledge Process Mining, Knowledge Process

Modelling and Pro-Active Knowledge Process

Support are integrated into highly innovative

application systems. Key results of the project are

dedicated software systems, which target the specific

needs of the case study partner, coming from

consulting, telecommunication and manufacturing

industries.

To determine the impact of the developed

software systems, a business benefit analysis is

currently conducted and based on the results,

respective business models will be developed

accordingly. Vague requirements for the approach

have been formulized at the beginning of the project,

which include

– the instantiation of a cost benefit matrix for

multilateral business benefits concepts,

– scenario and trend-based estimations of the eco-

nomic costs and benefits as well as

– the development of business cases which

compare the costs and benefits and provide an

assessment of the ACTIVE project results in the

context of the case study environments.

Against this background a reference framework

has been developed, which acts as a basis for the

approach on the business benefit analysis and

business model generation for the results of the

ACTIVE project and is illustrated in figure 1.

Business Benefit Analysis

-Scenario-based

-Trend-based

Business Model

Tools and

Methodologies

Case Study

Processes

Impact

Community

Impact

Processes

Impact

Technology

Component

Model

Task 4.1 &

Task 4.2

Involvement of

case study partner

Figure 1: Reference framework for the business benefit

analysis.

The framework describes the overall structure of

the analysis, including the involvement of the case

study partner and the considered existing tools and

methodologies. On a content level the foundation of

the framework are the elaborated case study

processes as well as the component ACTIVE model

and results from previously finished tasks, which

looked into costs and benefits of existing web 2.0

technologies. Process descriptions and component

model specify the developed solutions on a detailed

level and therefore allow a detailed business benefit

analysis which looks into the three pillars that cover

the whole area of influence of the developed

solutions within the case study partner:

– impact on technology,

– impact on processes,

– impact on community.

In addition to the reference framework, a course

of action has been agreed on which is based on the

three in figure 2.

During use cases and consists of three different

phases with, two sub-steps each. The course of

action is visualized the first phase, a cost-benefit-

framework is developed which aims to evaluate all

three pillars of the reference framework. The

A FRAMEWORK TO MEASURE THE IMPACT OF KNOWLEDGE PROCESS SUPPORTING TECHNOLOGIES - An

Integrated Approach for a Technology-oriented Business Benefit Analysis

255

development includes the identification of related

key aspects, covering the full scope of the pillars, as

well as the allocation of respective costs and benefit

factors to these key aspects and the definition of

qualified assessment methodologies. Ultimately the

cost-benefit framework should be highly scalable, so

that it can be dynamically adopted for the specific

business cases of the three case study partners.

Additionally, the framework is developed in a way

that it can easily be adopted for the evaluation of

other collaborative technologies.

Consolidating

the results

Framework

Gather

Specifications

Valuation

Preparing

The

Framework

Analysis

Collaborative

Proposal Writing

Collaborative

Bid Creation

Design Project

Planning

Data on costs

and benefits

•Expert Interview

•Calculations

Business-Model

•Workshop

•Profit Center

•Service

Cost Driver and

Benefits

•Key aspects

•Assessment

Business Model

Iteration

• Key Activities

• Costs/Revenue

Data on organ.

impact

• Expert Estimations

Business

Benefit Analysis

•Scenario-based

•Trend-based

a b

Figure 2: Selected course of action.

Consequently three distinctive and

complementing sub-categories have been identified

for each of the pillars, which are illustrated in figure

3 and represent the key aspects of ACTIVE

solutions. To measure the impact from a technology

perspective, IT-related costs and benefit factors have

to be considered. This is a rather traditional field

which includes the key aspects of IT investments, IT

adoption and IT administration. Evaluation of the

impact of collaborative technologies on company

processes can be achieved by looking into the key

aspects of business process support and training,

which are also kind of traditional, as well as

knowledge process support. The impact on

community is the most difficult of the three pillars.

It includes the key aspects knowledge management

with 2.0 technologies, information sharing with 2.0

technologies and use of context, which are difficult

to measure due to their supportive and therefore

more qualitative nature.

In a next step, relevant cost driver and benefits

which are considered relevant for the ACTIVE

solutions have been identified. The cost-benefit-

framework looks into these factors from three

different viewpoints: economic perspective, private

perspective and social perspective (Plum, 2008).

Additionally it differentiates between fixed and

variable (O’Sullivan and Sheffrin, 2007)(Garrison,

Noreen and Brewer, 2009) as well as direct and

indirect factors (Pietersz 2007)(Scott, 2003). The

underlying schema is displayed in figure 4.

Key Aspects

Impact on

Technology

Knowledge

Management with

2.0 Technologies

Knowledge

Process Support

Training

Use of Context

Investments

Impact on

Processes

Impact on

Community

Information

Sharing with 2.0

Technologies

Administration

Adoption

Business

Process Support

Figure 3: Cost-benefit framework.

Subsequently the identified cost and benefit

factors have been categorized into this schema and

integrated into the cost-benefit framework. Each

factor is allocated to at least one key aspect. As an

example, training costs is a fixed and direct

economic factor which can be allocated to the key

aspect training, while personal time saving is a

variable and indirect private factor which fits into

the key aspects knowledge process support and

business process support. Naturally economic

factors are more direct and quantitative while private

and social factors are more indirect and qualitative.

For the purpose of evaluating the factors of the

different key aspects, different methodologies have

been examined. For the traditional aspects IT

investment, IT adoption, IT administration, training

and business process support exist a plethora of

different tools and methodologies which can be

utilized to assess the economic impact (Pietsch,

2003). Relevant existing tools and methodologies

have been identified and will be adopted to match

the respective factors. Finally they will be integrated

into the framework to measure the traditional aspects

where possible.

However the impact of the four remaining, more

knowledge process related aspects, is difficult to

determine due to their qualitative and supportive

nature. Existing methodologies are not able to cover

KMIS 2010 - International Conference on Knowledge Management and Information Sharing

256

the full scope of these key aspects and can only be

partially integrated. Therefore the framework will be

complemented by dedicated, innovative

methodologies to assess these knowledge related

factors. For this purpose, approaches from other

fields will be examined and adopted, e.g. models

from the area of network theory will be used to

explain the influence of context on corporate

networks.

Cost Driver and Benefits

Frequency

Fixed

• Investments

• Time-related

Direct

• Quantitative

• Measurable

Variable

• Process-oriented

• Volume-related

Indirect

• Qualitative

•Immeasurable

Quality Characteristic

Economic

• Accountable

• more

quantitative

Private

•Focus on

individual

• mostly

qualitative factors

Social

•company equals

community

• mostly

qualitative factors

Figure 4: Classification of costs and benefits.

The first phase is completed by a first iteration

regarding the business models which will be

conducted together with the case study partners. The

business model canvas will be used, a tool that

allows to generate a high-level description of

business models in a workshop manner (Osterwalder

and Pigneur, 2009). The first iteration will look into

the key activities and resources, the cost structure

and the revenue streams of future business models,

which go along with the developed cost-benefit-

framework. The results of the workshop will

complement the framework and also be used for its

validation. Furthermore the relevance of the

different key aspects for each use case will be

determined in cooperation with the case study

partners and the respective factors will be adjusted

to their respective situation to generate three case

study specific cost-benefit-models. This especially

includes the assignment of the variable costs to the

respective process steps they are relevant for.

The second phase (Valuation Basis) is focused

on gathering the required data regarding the case

study specific cost-benefit-models and the overall

impact on the organization. Therefore expert

interview will be conducted and the gathered data

will be used as a basis for the following cost benefit

analysis. Where applicable the data-basis will be

complemented by adequate estimations on

respective, non-determinable factors.

Finally in the third phase (Analysis) a scenario-

and trend-based business benefit analysis will be

conducted for each case study use case using all

gathered data and pointing out the specific cost and

benefits not only from an economic, but also from a

personal and social perspective. On top of this,

business models will be finalized in workshops with

the case study partner. Again the business model

canvas will be used, utilizing the business benefit

analysis as value proposition.

4 CASE STUDY

In the scope of the ACTIVE project, different use

cases for the application of the developed

technologies have been identified for each of the

three case study partners coming from consulting,

telecommunication and manufacturing industries.

For the business benefit analysis, one use case for

each case study partner will be considered. The

selected use cases differ in their utilization of the

developed framework, setting a different focus on

the key aspects.

The consulting use incorporates collaborative

technologies for context-sensitive search and

browsing, therefore linking experts that work within

similar contexts and making common documents

accessible for each other. Based on this, a

collaborative proposal writing process has been

developed which targets the consultant’s ability to

react on incoming requests for proposals and quickly

mobilize the relevant resources. Easy and dynamic

access to relevant information is the deciding factor

for the success of the proposal, therefore a

collaborative proposal workspace is utilized which

provides swift access to knowledge and people. The

focus of the use case lies on the key factors IT

investments, IT adoption, IT administration,

training, business process support, information

sharing with 2.0 technologies and use of context

The manufacturing use case targets the planning

and execution of chip design projects and with

special focus on the reusability of existing work as

well as the sharing of experience between the

company’s design teams. The planning process is

supported by the semantic media wiki, which is an

advanced form of a media wiki allowing semantic

queries. It has been extended to visualize knowledge

processes, which is a necessary feature to share

information between teams. The design process

itself is a very complex procedure, consisting of a

A FRAMEWORK TO MEASURE THE IMPACT OF KNOWLEDGE PROCESS SUPPORTING TECHNOLOGIES - An

Integrated Approach for a Technology-oriented Business Benefit Analysis

257

series of formal, semi-formal and informal tasks

which are not predetermined. There is a strong

connection between the experience of a team and the

result of the process; therefore it is most valuable to

share knowledge between the design teams to get

optimal results. ACTIVE technology is utilized for

task mining, i.e. recording the series of steps from

each team, formalizing them into knowledge

structures and sharing them with other teams. The

use case doesn’t focus on the business processes

itself, but more on the results as well as the

formalization of the process. Therefore the important

key aspects are IT investments, IT adoption, IT

administration, knowledge management with 2.0

technologies, information sharing with 2.0

technologies and knowledge process support.

The telecommunication use case utilizes

collaborative technologies for collaborative bid

creation, i.e. to support their sales department in

dealing with incoming requests for bids. Time is a

critical factor within this use case; therefore it is

essential that sales people are able to access relevant

information, documents and people. The case study

partner utilizes so called activated applications,

which are frontend-tools like word or excel that have

been enabled by a plug-in to interact with the

ACTIVE knowledge workspace in the backend.

Information, knowledge processes and contacts are

gathered, processed and shared within the sales

team, therefore making best use of the collective

wisdom of the group. This use case covers the full

scope of the developed technologies; therefore all

key aspects have to be considered and assessed.

5 CONCLUSIONS

This paper presents an integrated approach to

evaluate the impact of knowledge process

supporting technologies. A reference framework is

introduced which defines the scope of the approach

and a respective course of action has been

introduced. Three areas of influence have been

identified and a cost-benefit-framework has been

developed, which describes these areas with nine

different key aspects. Cost drivers and benefits from

different fields have been integrated into this

framework and assessment methodologies for each

of them have been described.

Due to its high scalability and covered scope, the

introduced cost-benefit-framework seems to be very

promising. Additionally the framework seems to be

adoptable beyond the scope of the ACTIVE project.

However research is still ongoing and some open

questions still need to be answered. On the one hand,

the cost-benefit-framework has to be finalized,

gathering all relevant factors and categorizing them

into the existing schema. On the other hand,

methodologies for the evaluation of knowledge

process related aspects have to be finalized.

Therefore, existing methodologies from other areas

are currently examined regarding their relevance and

eventually need to be adopted for this specific case.

Finally the quality of this schema has to be validated

with real world data from the case study partner.

ACKNOWLEDGEMENTS

This work has been partly funded by the European

Commission through ICT Project ACTIVE:

“Advanced Context Technologies for collaboraTIVe

Enterprise” (No. ICT-FP7-215040). The authors wish

to acknowledge the Commission for their support. We

also wish to acknowledge our gratitude and

appreciation to all the ACTIVE project partners for

their contribution during the development of various

ideas and concepts presented in this paper.

REFERENCES

Hill, C., Yates, R., Jones, C., Kogan, S. L., 2006. Beyond

predictable workflows: Enhancing productivity in

artful business processes, IBM System Journal,

Volume 45, Number 4.

Brown, M., 2007. Social Computing Upends Past

Knowledge, Forrester Trends, March 8

th.

Forrester Big Idea, 2006. Social Computing, February 13

Active Knowledge-Powered Enterprise, www.active-

project.eu.

Ramirez, Y. W., Nembhard, D. A., 2004. Measuring

knowledge worker productivity, Journal of Intellectual

Capital, 5, 4.

Ahmed, P. K., Lim, K. K., Zairi, M., 1999. Measurement

practice for knowledge management, Journal of

Workplace Learning: Employee Counselling Today,

Vol. 11.

Pietsch, T., 2003. Bewertung von Informations- und

Kommunikationssystemen, Erich Schmidt Verlag, 2.

Auflage.

Schachner, L., 1973. Return on Investment: Its Values,

Determination and Uses, CPA Journal 43.

Haas, M., R., Hansen, M., T., 2007. Different knowledge,

different benefits: Towards a productivity perspective

on knowledge sharing in organizations, Strategic

Management Journal Vol. 28 Issue 11.

Eschenbach, S., Schauer, B., 2008. More Productive

Knowledge Work.

KMIS 2010 - International Conference on Knowledge Management and Information Sharing

258

Plum Consulting. 2008. A Framework for Evaluating the

Value of Next Generation Broadband.

www.plumconsulting.co.uk.

O’Sullivan, A., Sheffrin, S. M., 2007. Economics:

Principls in Action, Pearson Prentice Hall.

Garrison, R. H., Noreen, E. W., Brewer, P. C., 2009.

Managerial Accounting, McGraw-Hill/Irwin 13. ed.

Pietersz, G., 2007. Money Terms – Investment Reference,

Exposure Publishing.

Scott, D. L., 2003. Wall Street Words: An A to Z Guide to

Investment Terms for Today’s Investor, Houghton

Mifflin Company.

Osterwalder, A., Pigneur, Y., 2009. Business Model

Generation. Self Published.

A FRAMEWORK TO MEASURE THE IMPACT OF KNOWLEDGE PROCESS SUPPORTING TECHNOLOGIES - An

Integrated Approach for a Technology-oriented Business Benefit Analysis

259