THE INDIVIDUAL PERFORMANCE MEASUREMENT

FRAMEWORK IN VIRTUAL TEAM LEARNING

Virgilijus Sakalauskas and Dalia Kriksciuniene

Department of Informatics, Vilnius University, Muitines 8, 44280 Kaunas, Lithuania

Keywords: Distance learning, Virtual teamwork, Balanced scorecard concept, Test, Evaluation criteria.

Abstract: Learning in teams is aimed to develop skills of solving problem-based tasks by using various competences,

performing effective virtual communication and knowledge exchange activities. The problem analysed in

this article deals with the individual performance measurement for students working in virtual teams. The

evaluation and management of learning processes in virtual teams has many specific features, as compared

to the face-to-face learning mode. The goal of the article is to present evaluation framework, which could

allow us to estimate performance indicators, not only by measuring efforts of teams or individuals by

finding specific variables of virtual communication., but also by revealing their causal relationships to the

final outcome of the virtual project work. Therefore the suggested method is based on the balanced

scorecard concept, applied for virtual teamwork evaluation along analysed dimensions of the teamwork. The

balanced values are included to the compound grade for individual performance evaluation, which is

evaluation compounds consist of grades, estimated of individual and team input variables, and the results of

internal self-assessment of the virtual team members. The experimental data of the real virtual teamwork

was applied for analysis of the quality of suggested method, aimed to ensure that the final grade of the

project assessment reveals the individual progress of each learner related to team performance.

1 INTRODUCTION

Enterprises need workers with high level skills of

solving problems while acting in virtual teams,

formed for a particular project. Therefore such skills

are urgently needed for building students' future

career and should be effectively developed by

managing virtual team learning framework, of

improving knowledge exchange tools, development

communication environment and assessment system,

aimed to evaluation both performance of the team

and of the individual learner.

Learning in virtual teams creates need for

powerful learning environment, where skills of

cooperative work, and combining efforts towards

reaching learning goals and common tasks

fulfilment could be formed (Van Merrienboer &

Paas 2003). The teamwork processes can rarely be

evaluated by quantitative characteristics, except of

the time management and resource management

indicators. Performance measurement is important,

as it reveals negative and positive outcomes,

measures satisfaction and brings the result as

feedback to the system (Qiu et al, 2004). Yet, the

quality of the project result and of the teamwork

success can only be compared to the preset goals and

evaluated only after the final deadline of its

completion. Lack of possibilities to evaluate project

development and interrelatedness of the common

work bears problems, which are addressed in this

article: how to evaluate progress of the learner,

acting as the member of the team, and what

indicators could allow monitoring team project and

individual progress early since its starting phases.

In the following chapters we will present the

entity–relationships diagram of the virtual teamwork

learning environment, leading to the design of the

evaluation framework, based on concept of the

balanced scorecard. It integrates dimensions, factors

and indicators, applied for evaluation of the entities

of the virtual teamwork, by exploring their cause-

effect relationships as well. It is researched, how the

balanced scorecard approach can be applied for

designing framework of evaluating individual

learners. The calculated compound individual grade

integrates the tutors’ grades, and estimated efforts of

150

Sakalauskas V. and Kriksciuniene D. (2010).

THE INDIVIDUAL PERFORMANCE MEASUREMENT FRAMEWORK IN VIRTUAL TEAM LEARNING.

In Proceedings of the 2nd International Conference on Computer Supported Education, pages 150-155

DOI: 10.5220/0002806701500155

 SciTePress

the team, multiplied to the performance coefficient

of each member, joining individual measures and

evaluation grades. The experimental study presents

experiences of individual evaluation possibilities of

the teamwork, which involved 40 learners, arranged

in 7 teams, joining international students from 3

countries. The results and conclusions are

summarized in the final part of the article.

2 DIMENSIONS OF VIRTUAL

TEAM LEARNING

As the main goal of organizing team learning is to

prepare specialists for their career in contemporary

organizations, the tasks for learning should be

prepared according to their work specifics. The

organization of working teams are mostly related to

the project type of work for solving the problem,

acknowledged at the enterprise.

The problem-based learning is based on analysis

and solution of the real or prototype business case,

while traditional learning is based on finding a

solution of the unambiguous and defined task.

Therefore the organization and evaluation of team

learning is affected by specific conditions of

communication in virtual environment, sharing

responsibilities in groups, comprehension of specific

features of the problem-based task, and monitoring

possibilities of the learners, organized in virtual

teamwork project.

The understanding of virtual teamwork processes

can be revealed as the communication between the

following entities: tutor, team and individual

learners, and it is mostly expressed by the messages

of various types, submitted to the virtual

environment. The entity-relationship diagram of

problem-based learning in virtual teamwork is

presented in Fig.1., where the problem-based virtual

learning is supervised by the tutor. He presents the

learning curriculum and the undefined problem,

usually described as a business case which has to be

clarified and defined for the further solution within

the learning team, which has to discuss and set the

project goal according to the time resources and

competences of the team members.

The activities of the virtual team are monitored

by the tutor, who observes, adjusts and evaluates the

learning processes. The virtual teams consist of

members who play the assigned roles within the

team (leader, researcher, editor, etc.) In the virtual

environment there is no possibility to interact in the

face-to-face mode. Lack of this type of

communication is compensated by organizing the

video conferences for starting the learning session

and for presenting the final results. The main tool for

communication is writing messages in the virtual

environment. The variety of categories of the

messages is decided by the individual learners and

the team. They include online messages (online

chats) and offline messages in virtual environment

(forums).

The messages can be used for knowledge

exchange by presenting ideas, decision alternatives,

advices for the team to read common learning and

research materials, and also submitting the task,

assigned to the member by the team. They can

combine variety of tools for presenting information

in various digital formats.

The other type of messages can be used for

modelling communication within team, by

suggesting organizational information, motivating,

planning, discussing, voting activities, which jointly

create communication culture, tone and trust in

team.

The messages can be organized in thematic

threads, assigned for specific topics and for solving

intermediate tasks, leading to the project goal.

Figure 1: Entity-relationship diagram of learning in virtual

teamwork.

The traditional methods of evaluation are

basically oriented to grade the presented project

result, taking into account the activeness of the

individual learners and adjustments according to

THE INDIVIDUAL PERFORMANCE MEASUREMENT FRAMEWORK IN VIRTUAL TEAM LEARNING

151

self-evaluation of members within team. The

activeness component in the virtual team learning

environment can by mostly described as

“productivity” of the learner, mainly expressed by

the number of submitted messages. The assumption

that high productivity leads to high effectiveness and

good project result is not confirmed by any of the

experimental analysis or results of the self-

evaluation of the team members (Kriksciuniene,

Sakalauskas, 2008, 2009). In Fig.1 the main entities

which can be evaluated are: project goal (result),

teams, members, tasks and messages.

The main problem of evaluation is that all the

entities can be evaluated only after completion of the

project, when the adequateness of the achieved

result can be compared to the initial learning task.

Until the final deadline of the project the learner and

the tutor can not forecast the grade and the timely

completion of the project. One of the goals of the

design of the evaluation method is to apply

quantitative and qualitative measures for evaluation

and exploring their causal relationships leading to

learning strategy fulfilment and enabling tracking of

the progress. The balanced scorecard approach is

applied for modelling dimensions and factors

according to their direct and indirect influence to

final goals of the project performance Table 1.

The suggested approach makes parallel to the

balanced scorecard concept, presented by Kaplan

and Norton (1996).

Highest hierarchical dimension reflects the

traditional way to evaluate project, performed by the

team-final result and timely completion.

Generally, the dimensions, expressing team

performance and individual efforts tend to be

evaluated either by subjective measures or by

application easily observed indicators: duration of

presence in the virtual learning environment or

number of messages submitted to project forums.

The list of observed factors can be extended as in

Table 1. This scorecard can further be deployed to

the group and to individual dimensions, so that each

unit or person at the group knew his influence and

performance toward strategy fulfilment by team.

In Table 1 the virtual teamwork is affected by

the individual inputs, efforts, and role acting

dimensions. This means that analysis of individual

work can be understood as impulses, which either

bring the team nearer to its project goal, or

undermine it. The strength of factors of the

individual dimensions can be tracked by analysis of

messages. The dimension of messages bears factors

which not only provide communication statistics

about individuals and teams, but also helps to

analyse level of knowledge exchange, trust and

intensity of communication, which is finally

acknowledged by self-analysis of the team.

Computational analysis and factors of measurement

can allow to estimate the weight of the indicators,

according to their influence to project work, partially

automate the quantitative evaluation of work

processes of the virtual team by collecting

information, related to the derived indicators and to

track the performance of each member of the

learning team and evaluate their input to final result.

The evaluation model, based on mining the results of

experimental study, reflects cause-effect

relationships, which could affect the outcome of the

project work (Sakalauskas, Kriksciuniene, 2009).

Table 1: Project evaluation scorecard.

Compounds of balanced scorecard

Dimension Factors

Project goal (result)

dimension

Time limit met

Project goal achieved

Project report submitted

Presentation skills achieved

Teamwork experience gained

Virtual team

work dimension

Project problem defined

Project result augmented

Project quality calibrated

Project milestones met

Appropriate methods applied

Project phases documented

Project tasks shared in teams

Individual inputs

dimension

Assigned tasks fulfilled

Study materials understood

Participation in team events

Communication by using tools of

virtual environment

Individual efforts

dimension

Assignments admitted

Study materials analysed

Ideas generated and shared

Task performance discussed

Project processes organized

Decision making and evaluation of

alternatives

Individual role acting

dimension

Role accepted and played

Responsiveness to team tasks

Team motivation

Organizational efforts

Leadership

Messages analysis

dimension

Knowledge sharing intensity

“Richness” of explanation

Communication tools variety

Culture of communication

The traditional way of evaluation the final

outcome of the project by the final result and timely

CSEDU 2010 - 2nd International Conference on Computer Supported Education

152

completion does not provide possibility to equally

spread the workload and to forecast the final grade.

The individual evaluation measure is aimed to reveal

the ability of the individual learner to act as a team

member, show comprehension of the assigned tasks,

and to be able to organize his individual work in

alignment to teamwork processes.

Therefore in the Table 1 several areas of analysis

of the project teamwork were tightly related: time

management, assessing discussion quality both in

team and individually, performing knowledge

management, and making use of various virtual

collaboration tools. The evaluation compounds

include dimensions, tightly connected by causal

relationships. They include evaluation of teamwork

by its overall effectiveness, individual assessment

and self-evaluation inside team as well. The

important point of using these dimensions is aimed

to derive individual indicators, which could be

accumulated during the whole period of the project

and expressed in quantitative manner. The

accumulative individual evaluation can even serve to

motivate learners to increase their performance

during the project.

3 ASSESSMENT MODEL OF

VIRTUAL TEAMWORK

One of the main challenges is not only to calculate

statistics of communication processes, but to reveal

knowledge sharing gaps, problems, potentials of

individuals and team for achieving project goals. But

it appeared that the performance-based statistics

tools did not enable tutors to calculate the indicators

for measuring how the individual learner prepares

for the teamwork, what level of understanding of the

teaching materials he has achieved, and how he can

utilize his achieved level of knowledge by

submitting ideas.

In order to take into account the balanced

variables of the teamwork the compound grading

model for individual performance evaluation is

designed (Fig.2). The framework of evaluation

combines variables included into the entity-

relationship diagram (Fig.1) and project evaluation

scorecard (Table 1). The base for individual

evaluation is the team evaluation grade multiplied by

individual performance coefficient (average=1). The

individual grade is equal to the team grade if the

individual evaluation and self-evaluation measures,

consisting of corresponding factors, are equal to

average within the team. If the individual learner

outperforms the team average and individual added

value increases the project progress his individual

grade will be higher, than that of the teamwork.

These evaluation compounds consist of grades,

not only assigned by tutor, but also by estimating

individual and team input variables and the results of

internal self-assessment of the virtual team members

(Table 2). In table 2, the suggested indicators for

evaluation are presented. The teamwork grade is

assigned by the tutor after submitting project report

and making presentation of the project outcome.

These indicators are naturally applied in any project

setting. According to the project evaluation

scorecard (Table 1), the project goal dimension is

affected by the factors of the teamwork dimension.

These factors imply, that the teams who have better

values describing virtual teamwork dimension, get

better grade. The suggested weights for the team

evaluation grade are 60 % for project result and 40%

for the teamwork (Fig.2).

The individual evaluation coefficient requires

selecting and measuring variables which could

describe individual inputs, efforts and role acting

dimensions. The main factors, which could define

individual progress in these dimensions, are enlisted

in table 2. It is evident, that most of them can only

be described subjectively by applying qualitative

descriptions. In order to make the quantitative

evaluations, the dimension of messages is applied.

The factors are evaluated by assigning messages to

categories by their content, length, role of the author,

presence of ideas, and input to the final report. The

lowest dimension of messages gives us two main

advantages: they help to make numeric calculations

of grade compounds and help to track performance

during the project.

Self evaluation compounds can be selected by

the tutor. The members can award each other by

sharing 100 points along the three self-evaluation

indicators (Table 2). The provided indicators

Table 2: Evaluation compounds of the virtual teamwork.

Evaluation

compounds

Evaluation indicators

Team

work grade

Project work (timely completion, report,

presentation quality)

Team work (communication materials,

documents of project stages, the alignment of

the defined goals to team tasks)

Individual

evaluation

Inputs (messages with ideas, submitting tasks)

Efforts (messages discussion and organization

types, message length “richness”, response

time

Role acting

Self–

evaluation

Role fulfillment

Contribution to result

Team player

THE INDIVIDUAL PERFORMANCE MEASUREMENT FRAMEWORK IN VIRTUAL TEAM LEARNING

153

Figure 2: Framework of individual performance measurement.

had high cause-effect relationship to the individual

indicators in the experimental analysis (Sakalauskas,

Kriksciuniene, 2009).

The final measure for evaluation of individual

efforts is calculated by multiplying team grade by

the compound individual ratio (average=1).

Therefore individual grade is higher or lower than

the team result and can show, if the individual added

value increase or decrease the project progress (Fig.

2).

In the experimental research the weights applied

for individual evaluation, given by tutor and of the

self evaluation variables were 40 and 60 percent

respectively (Fig. 2). These values of weights were

selected by taking into account the analysed

relationships among the variables, but they can be

further explored by applying quantitative methods as

well. Application of the balanced approach revealed

the possibilities for enhancement functionality of

the virtual teamwork communication environment,

as all the calculations of factors and indicators could

be estimated half-manually and could be mainly

done only upon its completion, after applying

extensive procedures of data cleansing, normalizing,

converting, and processing.

4 THE EXPERIMENTAL STUDY

OF VIRTUAL TEAMWORK

ASSESSMENT

The experimental research explored the four-

compound model for evaluation process of

individual learners within teamwork (Fig.2). The

international group of students solved the common

task in the virtual environment during three weeks.

They were grouped in 7 teams with heterogeneous

nationalities, total 40 members, making project

solution of information systems model. Participants

were of 3 different specializations of studies:

Business informatics (Dresden University of

Technology), Business administration (St.Petersburg

state University), Business information systems

(Vilnius University).

During the project, there was no direct face-to-

face meeting. All communication was done in the

virtual environment. The team members were

assigned roles of leader, researcher, critics or

protocol writer. Student work was intensively

observed and tutored by lecturers of the three

universities.

The structure of the balanced scorecard model

was applied for exploring interrelated dimensions,

derived from the capabilities of the virtual

environment for quantitative analysis and

recommendations of participants of virtual projects

(Table 1). The experimental study had to reveal,

how the evaluation dimensions were interrelated and

influenced individual evaluation. The compound

grade according to the model of Fig. 2 and Table 2

was calculated and compared to the tutor’s grade.

The project result dimension was calculated as

average of grades (max =10), assigned by each tutor

of the seven teams. The grades were put for project

materials, prepared by the learners and submitted to

the common wiki space, and for the quality of the

final presentation. The data, recorded from virtual

communication, included total number of 9392

CSEDU 2010 - 2nd International Conference on Computer Supported Education

154

messages, consisting of 1215 offline messages and

8177 chat messages from online team conferences.

The variables for team and individual evaluation

were based on message analysis and derived

indicators. The analysis showed which indicators

were important to tutor, but undervalued by self-

analysis and vice versa. The correlation analysis of

the variables showed the strongest relationship

between tutors’ grade to the number of messages of

organizational type and submitting individual tasks.

Therefore, grade could not reveal the efforts of the

learner in the sphere of offering ideas or helping to

realize them inside the team. These characteristics

were better noticed by the team members and

expressed by self-evaluation characteristics. The

selected weight for self-evaluation was set to 60%.

The experimental research of the variables,

measured during the teamwork gave wider spread of

marks between the 40 individual learners, than by

applying conventional evaluation approach.

The project grade of the 7 teams varied from

6.536 to 9.786. The teamwork compound, which

involved factors of the virtual team work dimension,

was evaluated from 5.681 to 9.321. The individual

evaluation coefficient, consisting of inputs, efforts

and role acting components, weighted by 40% and

self-evaluation component (weight 60%) varied

among members from 0.514 to 1.444 (average=1).

The final individual marks of team members varied

from 0.292 to 1.33, and it allowed to conclude, that

the evaluation of the individual members of the

virtual team project by the suggested approach of

balanced evaluation revealed more sensitively the

efforts of the learners, than by sharing the project

result for all team and adjusting it by individual

grade, expressing active efforts of the learner.

The suggested evaluation method helped learners

to understand both his individual impact to the team

work and his acceptance by the team members and

lead to deriving final grade with the clearly

described quantitative measures of the four

compounds.

5 SUMMARY

The suggested method is based on the balanced

scorecard concept, applied for virtual teamwork

evaluation along analysed dimensions of the

teamwork. It allowed us to estimate performance

indicators, not only by measuring efforts of teams or

individuals by finding specific variables of virtual

communication., but also by revealing their causal

relationships to the final outcome of the virtual

project work. The balanced values are included to

the compound grade for individual performance

evaluation, which is designed by calculating the

teamwork grade and multiplying it to individual

performance coefficient.

These evaluation compounds consist of grades,

not only assigned by tutor, but also by estimating

individual and team input variables and the results of

internal self-assessment of the virtual team

members. The experimental data of the real virtual

teamwork was applied for analysis of the quality of

suggested method, aimed to ensure that the final

grade of the project assessment reveals the

individual progress of each learner related to team

performance.

The four compound approaches helps to solve

the difficult problem of forecasting of the project

outcome, as the team grade is put only after

completing whole project. As the individual

indicators are accumulated starting from the

beginning of the virtual project, they can encourage

learner to spread efforts during all project period for

getting both better individual and team grade. The

suggested assessment procedure leads to deriving

final grade with the clearly described quantitative

measures. The virtual team members' can realize his

impact to the team work and his effort acceptance by

the team members.

REFERENCES

Kaplan R. S., Norton D. P. (1996). Using the Balanced

Scorecard as a Strategic Management System //

Harvard Business Review., Jan-Feb. 1996.

Qiu, Y.F., Chui, Y.P., Helander, M.G. (2004).

Knowledge-Based Decision Making in Virtual Team

Environment. International Engineering Management

Conference 2004.

Sakalauskas, V., Kriksciuniene, D. (2008). Deriving

Knowledge Based Indicators For Monitoring Virtual

Project Teamwork. 20th International conference,

EURO MINI conference continuous optimization and

knowledge-based technologies, EUROPT'2008.

Sakalauskas, V., Kriksciuniene, D. (2009). Formation of

Team Working Skills by Applying Distance Learning

Methods. Proceedings of the 1st International

Conference on Computer Supported Education

(CSEDU 2009), Vol. 2, p. 131-135.

Van Merrienboer, J. J. G., & Paas, F. (2003). Powerful

learning and the many faces of instructional design:

towards a framework for the design of powerful

learning environments. In E. De Corte, L. Verschaffel,

N. Enstwistle, & J. J. G. Van Merrienboer (Eds.),

Powerful learning environments: Unravelling basic

computers and dimensions. Oxford: Elsevier Science.

THE INDIVIDUAL PERFORMANCE MEASUREMENT FRAMEWORK IN VIRTUAL TEAM LEARNING

155