Experience from the WELKOM Project
Maurice Grinberg and Evgenia Hristova
Central and Eastern European Center for Cognitive Science, New Bulgarian University
21, Montevido street, Sofia, Bulgaria
Keywords: E-Learning, usability, learning styles.
Abstract: The main features of an integrated Computer Assisted Training (CAT) system design and deployment
methodology, developed in the WELKOM project, are summarized and discussed. The two main
dimensions of the methodology – educational efficiency and usability – are presented with a focus on the
latter. Selected results from usability and learning style testing during the application of the methodology in
the development of three different CAT systems in three different educational contexts (a factory, a SAP
training company and an university) are presented and the results are discussed with regard to their impact
on the implementation process and to the applicability of the integrated methodology as a whole.
The WELKOM project, financed by the Leonardo
program of EC, has as a main goal the development
and application of an integrated methodology for
deployment of Computer Assisted Training (CAT)
systems. The methodology has been developed by
integration of various methods used in the field of
instructional design, knowledge assessment, and
usability research. In the course of the project, the
methodology is being applied in the deployment of
three CAT systems: two CAT systems in business
companies and one in a university. The CAT
systems differ in various aspects such as educational
content, the purpose of training, the technical
solutions. The application of the methodology must
lead to optimized deployment time, better
educational achievements and decreased training
The main purpose of the Integrated Deployment
Methodology (IDM) is to apply various methods in a
coherent and complementary way in order to
optimize a CAT system in the shortest time. An
important research objective of the project is to test
and evaluate the methods used and investigate their
applicability, efficiency, and usefulness. Here, we
report the first results of the application of IDM
described in the preceding sections for three CAT
systems. These three CAT systems differ in many
respects and are a good test bed for IDM. They are
developed, used, tested, and improved during the
WELKOM project. The three systems differ in their
educational goals, educational content, and software
implementation. The three CAT systems will be
further denoted to as CATS-1, CATS-2, and CATS-3.
CATS-1 was developed in TURBOMECA,
which is a French company working in the engines
industry and was aimed at training recent recruits
occupying various jobs in various departments at the
company. The scope of CATS-1 was to provide
training in the use of the company intranet
information systems. The trainees have diverse
backgrounds and are employed for positions at very
different levels. CATS-1 is built on the basis of an
existing CAT systems already used in the company.
CATS-2 is being developed for a Bulgarian
company providing SAP training for accountants
and financial managers. The learners are from
different companies and typically are highly
qualified specialists. CATS-2 has been built up from
scratch during the project by one of the partners –
ISTAR training. In this case, all the element of IDM
could be used and tested and most of the
recommendation taken into account.
CATS-3 implements a course in assessment
methods for university professors in the New
Grinberg M. and Hristova E. (2007).
In Proceedings of the Third International Conference on Web Information Systems and Technologies - Society, e-Business and e-Government /
e-Learning, pages 545-548
DOI: 10.5220/0001287005450548
Bulgarian University. An open source e-learning
environment (Moodle, has been
used. The open source allows modification of the
system in every aspect – it could be modified to suit
the particular needs of the users. Such modifications
are done in accordance with IDM application results.
The methodology proposed is integrating methods
for assuring both – educational efficiency and
Educational efficiency is generally considered as
the most important in training. However, when we
consider CAT, the educational efficiency is closely
tied to the delivery methods usually based on
computer environments, multimedia, web-based
platforms etc. The latter give much more options
than constraints in training but are related to many
usability issues. If the latter are unresolved, the
system becomes difficult to use by the trainees. In
such a way instead accelerating learning, CAT can
be an obstacle to it especially for trainees with no or
little experience with computers and internet.
In order to maximize the utility of each method
the deployment process is divided in several phases:
analysis, design, development, and implementation.
Throughout all phases tests for assuring both
educational efficiency and usability are performed.
The process is iterated and the phases are repeated at
least partially until the training goals are met,
including the time and effort constraints.
In the framework of the WELKOM project the
usability is tested and assured by applying several
methods from different perspectives. On one hand,
tests are performed from the experts’ perspective –
in such a way we take advantage from the
accumulated knowledge in the domain. On the other
hand, tests are performed which give also the users’
perspective. In such a way the CAT system is
meeting the special requirements, abilities, learning
styles and expectations of the learners.
In this phase the technical basis of the CAT system
is chosen and the initial design is being created. The
platforms and their options are compared with the
specific course requirements. The comparison is
based on the educational content and the trainees’
learning styles and preferences.
In order to explore the learners’ attitudes and
preferences, three questionnaires were used which
are presented in the following sections.
The first questionnaire is aimed at exploring the
trainees’ attitudes towards computers. Individuals’
attitude towards computers is a key component for
the understanding of the acceptance of, learning
success and satisfaction with CAT systems. The
questionnaire used was developed by M. Paprzycki
and D. Vidakovic (see Paprzycki et al., 1995). The
rationale behind its use is that different target groups
could have different attitudes ranging from fear and
confusion to pleasure and amusement, or the
computer might be perceived just as a tool for doing
the job. On the basis of the prevailing attitudes,
choices about the system design can be made. For
instance, the choices may be between a ‘formal’
CAT system or one with funny elements or even
using a game-like format.
The questionnaire for the attitude to computers
was administered during the development of CATS-
2. As expected, all of the trainees showed very
positive attitudes as all of them were using
computers at their work.
Recommendations concerning the CAT system
design were focused on functionality and
clarification of the actions that could be performed.
The second questionnaire used is David Kolb’s
Learning Style Inventory (LSI). This questionnaire
provides a framework for identifying students'
learning style preferences. The model postulates two
modes of getting experience – Concrete Experience
(CE) and Abstract Conceptualization (AC) – and
two modes of transforming experience – Reflective
Observation (RO) and Active Experimentation (AE)
(Kolb, 1984; Kolb, 2005; Kolb & Kolb, 2005). On
the basis of these two dimensions, four learning
styles can be characterized – converging (common
sense), dynamic, imaginative, and analytic.
Determining trainees’ learning styles helps in
designing the course format and especially how the
material should be presented – e.g. should the
material be presented in theoretical lectures or by
using case studies and exercises.
WEBIST 2007 - International Conference on Web Information Systems and Technologies
LSI was administered to the prospect trainees
who were supposed to use CATS-1, CATS-2, and
CATS-3. For CATS-2 the predominant modes of
learning are abstract conceptualization and active
experimentation (see Figure 1). Most of the trainees
belong to the converging type. They prefer technical
tasks, and are less concerned with people and
interpersonal aspects. People with converging style,
like to experiment with new ideas and to work with
practical applications. These learners like following
detailed sequential steps, hands-on activities, trial
and error, and being given clear objectives with a
logical sequence to activities. They learn through
interaction and thus computer-based learning is
more effective with them than other methods. Some
of the trainees belong to the analytic type. They like
abstract ideas and concepts, conceptual models,
designing experiments, reading, theories, and
structured activities. They enjoy a systematic
approach, detailed directions, and computer assisted
Figure 1: LSI results involving 21 learners using CATS-2.
On the basis of these results the following
conclusions and recommendations concerning
CATS-2 were made. The training should focus on
thinking, not on feeling; clear, logical conceptual
schemes without details and examples should be
used; the trainees should have the opportunities to
solve problems and find solutions; more practical
asks should be included in the course.
The third questionnaire that was administered is the
VARK questionnaire assessing learner’s preferred
mode of learning and learners’ preferences for the
way they work with information. The four modes
considered are visual, auditory, reading, and
kinaesthetic. As a result of the test we can determine
the preferred mode of information processing of the
trainees. Knowing trainees’ learning styles
contribute to improve learning through improving
the information presentation mode. For example, if
most of the people prefer auditory presentation of
the information, such an option should be provided
in the systems (Fleming & Mills, 1992; Fleming,
The VARK questionnaire was administered for
CATS-1, CATS-2, and CATS-3. Results for CATS-
2 are presented here. The results show that most of
the learners’ possess a multimodal style. The
distribution is shown in Figure 2.
The main recommendations made were that
there should be a lot of practical exercises. The
material to be learned should be presented as a text.
Auditory presentation of the material is also possible
(however, this is not mandatory and if such mode is
provided, an option for switching off the auditory
presentation of the material should be easily
R; 10
K; 14
V; 2
A; 7
Figure 2: VARK test results for 21 learners using CATS-2.
The total number is greater than 21 as some of the learners
have more than one strongly manifested style.
In this phase the actual development of the CAT
system takes place.
In the IDM three different methods that provide
usability information for the development phase are
proposed: heuristic evaluation, user testing, and eye-
tracking recordings.
The heuristic evaluation of the system is
performed by trained experts. It is based on well-
defined and broadly accepted usability guidelines.
The aim of evaluation is to ensure that the CAT
system is built in a way that conforms to usability
standards and that information is presented in a
manner that maximizes its educational value As a
result of the evaluation, detailed recommendations
of the improvements of the system are provided
(more detailed description of the heuristic evaluation
method can be found in Nielsen, 1994).
The heuristic evaluation done during the
WELKOM project is made on the basis of 86 rules
divided in several categories. Heuristic evaluations
have been used for all three CAT systems. CATS-2
again will be presented here as an illustration. In the
beginning, two CAT prototypes were tested.
from the WELKOM project
Separate heuristic evaluations were performed with
both prototypes. Problems that were identified were
summarized in reports and presented to the
developers of the system. After consideration of the
severity of the problems on one hand and of the
technical possibilities on the other, one of the
prototypes was chosen. After that an additional
heuristic evaluation of the selected prototype was
performed and again the new recommendations were
implemented. This was done in several iterations.
The second method proposed is user testing
(Rubin, 2001; Lewis & Rieman, 1994; Nielsen &
Mack, 1994; Kunyavski, 2003; Dumas & Redish,
1999). In this type of study, representatives of the
trainees are asked to perform specified tasks with the
CAT system. Their actions and comments are
recorded and analyzed. It is important to test the
CAT system with real users, as neither the designers,
nor the usability experts can foresee all the problems
that the users could have in a task completion. The
difficulties experienced by the users are analyzed
and recommendations for the CAT system
improvement are given
User testing has been performed so far for
CATS-1 and CATS-3. For CATS-1, ten
representative tasks were selected (e.g., ‘Start the
system and log on’, ‘Start/stop/pause the lesson’,
‘Take a test’, and ‘Look at the test results’). The user
testing identified many additional problems in
comparison with the heuristic evaluation some of
which were crucial for the efficient work with the
CAT systems.
IDM has been applied so far up to the
implementation phase in which presently is only
CATS-1. This phase is actually the most interesting
and intense in terms of tests both for educational
efficiency and usability. A very important new
element during this phase will be the achievements
assessment which will be the basis for evaluating the
meeting of goal knowledge level.
Although no full iteration in IDM has been
performed so far, it can be said that combining
several methods during all the phases of CAT
deployment gives a lot of useful complementary
information that puts together the efforts of the
training stake-holders, the IDM team, the software
developers, and the educational content providers.
Thus IDM seems to allows not only the building of
an optimized CAT system but leads to economy of
time and efforts by making the prevention and the
solution of problems at the most appropriate
moments of the CAT development. Although the use
of so many tests might seem quite expensive the
authors believe that after the optimization of IDM at
the end of the project the final IDM methodology
will prove to be very efficient in terms of ROI. This
claim, however, is a research question which will be
dealt with in the remaining decisive one year of the
All the work presented in this paper was supproted
by the WELKOM project. We would like to
acknowledge the fruitful discussions and
collaboration with the WELKOM partners and the
NBU team.
Dumas, J. & Redish, J., (1999), A Practical Guide to
Usability Testing. Portland, OR: Intellect.
Fleming, N.D. & Mills, C. (1992). Not Another Inventory,
Rather a Catalyst for Reflection. To Improve the
Academy, 11, 137-155.
Fleming N. D. (2001) Teaching and Learning Styles:
VARK Strategies Honolulu Community College.
Kuniavsky, M. (2003). Observing the user experience: A
practitioner's guide to user research. Morgan
Kaufmann, San Francisco, CA.
Kolb, D. A. (1984). Experiential Learning: experience as
the source of learning and development. New Jersey,
Prentice Hall.
Kolb, D. A. (2005). The Learning Style Inventory Version
3.1. Boston, MA Hay Resources Direct.
Kolb, A. Y. & Kolb, D. A. (2005), The Kolb Learning
Style Inventory- Version 3.1. Technical Specifications.
Experience Based Learning Systems, Inc., Case
Western Reserve University.
Nielsen, J. & Mack, R.L. (Eds) (1994). Usability
Inspection Methods, . John Wiley & Sons, New York,
Lewis, C. & Rieman, J. (1994). Testing the design with
users. In: Task-Centered User Interface
Design.Retrieved November, 13, 2006 from
Paprzycki, M., Vidakovic & Ubermanowicz, S. (1995).
Comparing Attitudes Toward Computers of Polish and
Americam Prospective Teachers. Technology and
Teacher Education Annual, AACE, Charlottesville,
VA, 45-48
Rubin, J. (2001) Handbook of Usability Testing: How to
Plan, Design, and Conduct Effective Tests. John Wiley
& Sons.
WEBIST 2007 - International Conference on Web Information Systems and Technologies