EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE

AND CO-OPERATION PLATFORMS

Claudia Loroff

Fraunhofer Institute for Software and Systems Engineering (ISST), Berlin

Keywords: Evaluation concept, knowledge management s

ystems, CSCW systems, integrated knowledge and co-

operation platforms

Abstract: This article introduces a concept for evaluating integrated knowledge and co-operation platforms which was

derived from systematic examination of computer supported co-operative work (CSCW) and knowledge

management systems and from research of available evaluation approaches to CSCW and knowledge

management systems. It consists of various evaluation perspectives (individual, group, organisation,

environment and technical system), thereby introducing comprehensive objectives, specifying topics and

exemplary items for these perspectives. Considering experiences made with this concept, potential

implementation scenarios are introduced.

1 INTRODUCTION

The impact of so-called knowledge work on

enterprises and organisations is increasing. Merging

CSCW and knowledge management systems in an

integrated knowledge and co-operation platform can

technically support the process of knowledge work

in a comprehensive way (Fuchs-Kittowski and

Reuter 2002). The value of any technical support can

be determined by evaluation (Englberger 2001). As

the idea of an integrated knowledge and co-

operation platform is new, there is no evaluation

concept available at present. Former concepts

regarded either CSCW or knowledge management

systems.

This article introduces a concept for evaluation

f integrated knowledge and cooperation platforms

derived from systematic examination of computer

supported co-operative work (CSCW) and

knowledge management systems and from research

of available evaluation approaches to CSCW and

knowledge management systems. Based on this

approach different evaluation perspectives are

deduced which are described by comprehensive

surveys and specifying topics.

2 DEVELOPING THE CONCEPT

The first step in developing the concept was to

approach CSCW and knowledge management

systems systematically. As a result, the following

working definitions for CSCW and knowledge

management systems were specified:

– CSCW systems consist of computer technologies

for supporting any kind of communication,

cooperation, and coordination in groups.

– Knowledge management systems support

varying processes of generating, gathering,

securing, transmitting and utilising knowledge

for the benefit of an organisation.

Moreover, similarities, differences

, aims, and

supporting technologies of both system classes were

identified. In CSCW systems work on the object is

at the centre. Thereby, group work independent from

time and space shall be facilitated (see Teufel et al.

1995, Krcmar and Schwabe 1996b). In knowledge

management systems, knowledge of the object is in

the focus. Technical support is meant to make

knowledge in an enterprise easy to communicate,

recyclable and expandable.

Evaluations of knowledge management systems

ostly examine the availability of technical support

for certain knowledge management strategies, and

resp. how often these are applied by users. These are

predominantly easily quantifiable aspects that have

been determined in research (e.g. Zack 1999).

176

Loroff C. (2005).

EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS.

In Proceedings of the Seventh International Conference on Enterprise Information Systems, pages 176-179

DOI: 10.5220/0002551601760179

 SciTePress

Further evaluations of CSCW and knowledge

management systems stress the comparison of

systems and their support functions (Stevens and

Scholtz 1999, Cobos et al. 2002, Mueller-Prothmann

and Siedentopf 2003). It is generally apparent that

there are only a few comprehensive concepts for

evaluating knowledge management systems (Maier

and Haedrich 2001, Quek and Shah 2004) and

CSCW systems (Ramage 1999). As a result of the

analysis of evaluation approaches for CSCW and

knowledge management systems deficiencies could

be found in the support of unpredictable, creative or

social processes and the impact an individual, a

group, the organisation or the environment/setting

has on a CSCW or knowledge management system.

In addition, there is still a lack of evaluation

concepts explicitly designed for integrated

knowledge and cooperation platforms. Based on the

analysis of CSCW and knowledge management

systems and the findings in scientific references, an

evaluation concept for integrated platforms was

deduced and will be introduced in the following part.

3 DESCRIPTION OF THE

EVALUATION CONCEPT

The evaluation concept introduced here regards five

perspectives:

– The individual perspective, i.e. an individual

systems user (employee),

– The group perspective including communities,

– The organisational perspective, where the system

is (partly) applied,

– The environmental perspective, i.e. the relevant

world beyond the organisation,

– The system perspective, i.e. the integrated

knowledge and co-operation platform.

This subdivision is based on the following

consideration: individuals can form groups. They are

employees. Groups consist of employees of an

organisation. This organisation is embedded in an

environment. The technical system (platform) is

provided to some or all individuals/groups of the

organisation and thus penetrates the other

perspectives. For this reason, some items can for

example be attributed to the individual perspective

and/or, according to focus, to the system

perspective.

The evaluation concept provides every

perspective with comprehensive tasks, specifying

topics and exemplary items. Generating an

exhaustive list of items is not sensible as they must

always be depending on the evaluated system, the

evaluation context and the requirements of the

evaluating person.

The perspectives individual, group, organisation

and environment are similar in structure. The

comprehensive tasks refer to the respective

interdependency with the system. They are specified

for every perspective by identical topics. The topic

items differ from perspective to perspective. The

group perspective was additionally supplied with the

topic “founding a group” as groups are at the centre

of interests in an integrated platform. The system

perspective that all other perspectives refer to has a

different structure. Here are predominantly system-

describing components more important than

interaction with the system.

Table 1 shows the exemplary structure of the

evaluation concept with the perspective group. The

entire structure is shown in Table 2.

Table 1: Implementation of the Group Perspective

Task Topic Indications for generating

items

Improvement Have suggestions for

improving group work been

generated and introduced to

the system? What do they

refer to?

Which impact does a group

have on the system?

acceptance/

motivation for

application

To which extent is the system

utilised for group work if

there are occasions for it?

How is the system accepted

by groups?

facilitation of

group

formation

How far is self-directed and

task-related formation of

groups facilitated?

effect by

utilising the

system on the

work per se

Which are the precise

objectives of the groups when

applying such a system for

their collaboration, and how

are these regarded by the

system?

handling of

knowledge

and know-

ledge gain

Which system features were

used in the group for

generating new knowledge

and how does the system

motivate users to

preserve/secure it?

Which effects does the system have on the group?

support of

social

processes

How strong is the trust in

other group members?

EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS

177

Table 2: Structure of the Evaluation Concept for integrated Knowledge and Co-Operation Platforms

Perspective Comprehensive Task Specifying Topics

Improvement Which impact has the individual/

group/organisation/environment on the system?

Acceptance and motivation for usage

Impact of usage on work per se

Handling of knowledge and knowledge gain

Support of social processes

Individual

Group

Organisation

Environment

Which impact has the system on the

individual/group/organisation/environment?

Support of group formation (only group)

Development phase Which is the system status?

Available functions

Independent selection of tools

Independent organisation of knowledge/contents

Can the system be adjusted to specific user needs,

habits and wishes?

Configurable message function

Codification

Socialisation

Personalisation

How does the system support different

knowledge management strategies?

Combination of approaches

Does the system support creative processes? Support of creativity by tools

Integration and organisation of knowledge Does the system support unpredictable processes?

Integration and organisation of or with persons

Available tools Which links of the system with tools available to

users are there?

Integration of available tools

Utility

Usability

Technical

System

What are the characteristics of the functions and

how are they dealt with?

Application of system by users

4 EXPERIENCES

During the project „Knowledge co-production in

knowledge-intense services“ (German abbr.: WiKo),

the Fraunhofer Institute for Software and Systems

Engineering Berlin in collaboration with the

Fraunhofer Institute for Applied Information

Technologies developed an integrated knowledge

and co-operation platform for supporting interaction-

oriented knowledge management. In the test phase

of the prototype valuable experiences with the

evaluation concept mere gathered. Two major

approaches of putting the concept into practice were

observed:

Top-down: Hypotheses are deduced from a

given task. In doing so, relevant perspectives,

questions and topics for these perspectives are taken

into consideration. On this basis, the hypotheses are

substantiated. Defining relevant perspectives is

important in this process. Reducing the complexity

of the hypotheses to a maximum of two perspectives

for consideration proved sensible. The system

perspective in combination with one of the other

perspectives was the common case. A considerate

selection of topics allowed a fast and easy

generation of corresponding items. In approaching

step by step, the hypothesis as well as the

corresponding questions can be looked at from

different angles. This allows for generating items

comprehensively and for emphasising certain

aspects in a considered, conscious and well-founded

way.

Bottom-up: In developing survey instruments,

the evaluating persons appeared to having had

considered certain items in advance that seemed to

be of higher interest or should generally be regarded

in an evaluation. Our evaluation system allows for

bringing such items into a context. They are

allocated to a perspective and a related topic. A

further step deals with analysis of the topics

relevance for the evaluation. If the topic is relevant,

further items can be generated. If the topic is

irrelevant, the items need not be regarded.

In analysing and interpreting data, perspectives

and topics help continuingly focussing on vital

aspects, affirm or disprove hypotheses.

Summarising, with the help of perspectives,

questions and topics the survey results can be

illustrated and demand for further evaluation or

further development of the system considered may

be shown. The contents of the system perspective

provide essential questions and topics for developing

integrated knowledge and co-operation platforms

and for iterative development processes. At the time

of introducing the prototype for practical work or

platforms in operation contents of all perspectives

must be regarded. Comparing platforms, they should

ICEIS 2005 - HUMAN-COMPUTER INTERACTION

178

be done on the basis of questions and topics of the

system perspective. If there are practical experiences

at hand, all other perspectives should also be

compared for drawing comprehensive conclusions.

No evaluation considers only the questions and

corresponding topics described above. Further

aspects influencing the results also need to be

regarded. Such influences may be survey

instruments, sample surveys, the setting for the

evaluation, available time, technical and financial

resources. Bringing these factors of influence into

consideration, the application of the evaluation

concept portrayed here leads to a systematic and

standardised approach to knowledge and co-

operation platforms.

5 DISCUSSION

It was proved that the concept supports the

following phases in evaluating integrated knowledge

and co-operation platforms: the generation of

questions, hypotheses and items, whereby a top-

down respectively a bottom-up approach could be

distinguished, furthermore data analysis, the

interpretation of data, the allocation of results into a

general concept, and the deduction of future

proceedings. Applying this concept allows the

comprehensive systematic comparison of different

platforms. Unfortunately, this could not be realised

so far. Furthermore, the concept entails different

opportunities for extension. It was already pointed

out that the system perspective influences all other

perspectives. The interaction of the system with all

other perspectives is continuously brought up. The

interaction amongst the other perspectives was

explicitly not regarded for not making the evaluation

concept too complex. An extension would be a

sensible addition to the concept. A further extension

could be the development of particular items for

single topics of the perspectives. On one hand, this

would result in higher standardisation of the

approach and better comparability of results. On the

other hand, it would lead to a loss of flexibility of

the concept. This openness allows for putting

different emphasis and for adding special cases

depending on the evaluation context. Putting the

concept into action motivates to view the evaluation

of the platform from different angles. This supports

the generation of new ideas and broadens

understanding.

An evaluation concept can only provide the

evaluating person with an appropriate setting if

he/she is able to properly classify his/her own

objectives and relevant objectives of other persons,

of groups, of the organisation or even of the

environment. Predictable and unpredictable frame

conditions have a great impact on the operation and

the results of any evaluation. The evaluation concept

must therefore be flexible. The deduced perspectives

individual, group, organisation, environment and

system can each be applied individually for

generating relevant data; but grasping the platform

to be evaluated in its entirety it is necessary and

advisable to include all perspectives in the

evaluation.

REFERENCES

Cobos, R.; Esquivel, J.; Alamán, X., 2002. IT Tools for

Knowledge Management: A Study of the Current

Situation. UPGRADE, Vol. 3, Nr. 1, P. 60-71.

Englberger, H., 2001. Evaluierung. In: Schwabe, G.;

Streitz, N.; Unland, R. (Hrsg.): CSCW-Kompendium.

Heidelberg: Springer Verlag.

Fuchs-Kittowski, F.; Reuter, P., 2002. E-Collaboration für

wissensintensive Dienstleistungen. Information

Management & Consulting, Bd. 17, Nr. 4, P. 64-71.

Krcmar, H.; Schwabe, G., 1996b. CSCW-Werkzeuge.

Wirtschaftsinformatik, Nr. 38, P. 205-225.

Maier, R.; Hädrich, T., 2001. Modell für Erfolgsmessung

von Wissensmanagementsystemen.

Wirtschaftsinformatik, Bd. 5, Nr. 43, P. 497-509.

Mueller-Prothmann, T.; Siedentopf, C., 2003. Designing

Online Knowledge Communities: Developing a

Usability Evaluation Criteria Catalogue. In:

http://www.knowledgeboard.com/download/2760/kms

s03_03.pdf. Last visit: 25.5.04.

Quek, A.; Shah, H., 2004. A Comparative Survey of

Activity-Based Methods for Information Systems

Development. In: Seruca, I.; Filipe, J.; Hammoudi, S.;

Cordeiro, J. (Eds.): 6th International Conference on

Enterprise Information Systems CiCEIS, Porto,

Portugal, 2004. Vol. 5, P. 221-229.

Ramage, M., 1999. The Learning Way. University of

Lancaster. In: http://systems.open.ac.uk/page.cfm?

pageid=MagnusRthesis. Last visit: 3.7.04

Stevens, M.; Scholtz, J., 1999. Modified Field Studies for

CSCW Systems. In: http://www.nist.gov/itl/iad/

IADpapers/modified_cscw.pdf. Last visit: 6.6.04.

Teufel, S.; Sauter, C.; Mühlherr, T.; Bauknecht, K., 1995.

Computerunterstützung für die Gruppenarbeit. Bonn:

Addison-Wesley Verlag.

Zack, M. H., 1999. Managing Codified Knowledge. Sloan

Management Review, Vol. 40, No. 4, P. 45-58.

EVALUATION CONCEPT FOR INTEGRATED KNOWLEDGE AND CO-OPERATION PLATFORMS

179