THE ROLE OF USABILITY TESTING IN AN INTEGRATED

APPROACH TO CAT DEPLOYMENT

Experience from the WELKOM Project

Maurice Grinberg and Evgenia Hristova

Central and Eastern European Center for Cognitive Science, New Bulgarian University

21, Montevido street, Sofia, Bulgaria

Keywords: E-Learning, usability, learning styles.

Abstract: The main features of an integrated Computer Assisted Training (CAT) system design and deployment

methodology, developed in the WELKOM project, are summarized and discussed. The two main

dimensions of the methodology – educational efficiency and usability – are presented with a focus on the

latter. Selected results from usability and learning style testing during the application of the methodology in

the development of three different CAT systems in three different educational contexts (a factory, a SAP

training company and an university) are presented and the results are discussed with regard to their impact

on the implementation process and to the applicability of the integrated methodology as a whole.

1 PROJECT GOALS

The WELKOM project, financed by the Leonardo

program of EC, has as a main goal the development

and application of an integrated methodology for

deployment of Computer Assisted Training (CAT)

systems. The methodology has been developed by

integration of various methods used in the field of

instructional design, knowledge assessment, and

usability research. In the course of the project, the

methodology is being applied in the deployment of

three CAT systems: two CAT systems in business

companies and one in a university. The CAT

systems differ in various aspects such as educational

content, the purpose of training, the technical

solutions. The application of the methodology must

lead to optimized deployment time, better

educational achievements and decreased training

time.

The main purpose of the Integrated Deployment

Methodology (IDM) is to apply various methods in a

coherent and complementary way in order to

optimize a CAT system in the shortest time. An

important research objective of the project is to test

and evaluate the methods used and investigate their

applicability, efficiency, and usefulness. Here, we

report the first results of the application of IDM

described in the preceding sections for three CAT

systems. These three CAT systems differ in many

respects and are a good test bed for IDM. They are

developed, used, tested, and improved during the

WELKOM project. The three systems differ in their

educational goals, educational content, and software

implementation. The three CAT systems will be

further denoted to as CATS-1, CATS-2, and CATS-3.

CATS-1 was developed in TURBOMECA,

which is a French company working in the engines

industry and was aimed at training recent recruits

occupying various jobs in various departments at the

company. The scope of CATS-1 was to provide

training in the use of the company intranet

information systems. The trainees have diverse

backgrounds and are employed for positions at very

different levels. CATS-1 is built on the basis of an

existing CAT systems already used in the company.

CATS-2 is being developed for a Bulgarian

company providing SAP training for accountants

and financial managers. The learners are from

different companies and typically are highly

qualified specialists. CATS-2 has been built up from

scratch during the project by one of the partners –

ISTAR training. In this case, all the element of IDM

could be used and tested and most of the

recommendation taken into account.

CATS-3 implements a course in assessment

methods for university professors in the New

545

Grinberg M. and Hristova E. (2007).

THE ROLE OF USABILITY TESTING IN AN INTEGRATED APPROACH TO CAT DEPLOYMENT - Experience from the WELKOM Project.

In Proceedings of the Third International Conference on Web Information Systems and Technologies - Society, e-Business and e-Government /

e-Learning, pages 545-548

DOI: 10.5220/0001287005450548

 SciTePress

Bulgarian University. An open source e-learning

environment (Moodle, www.moodle.org) has been

used. The open source allows modification of the

system in every aspect – it could be modified to suit

the particular needs of the users. Such modifications

are done in accordance with IDM application results.

2 INTEGRATED CAT

DEPLOYMENT METHDOLOGY

The methodology proposed is integrating methods

for assuring both – educational efficiency and

usability.

Educational efficiency is generally considered as

the most important in training. However, when we

consider CAT, the educational efficiency is closely

tied to the delivery methods usually based on

computer environments, multimedia, web-based

platforms etc. The latter give much more options

than constraints in training but are related to many

usability issues. If the latter are unresolved, the

system becomes difficult to use by the trainees. In

such a way instead accelerating learning, CAT can

be an obstacle to it especially for trainees with no or

little experience with computers and internet.

In order to maximize the utility of each method

the deployment process is divided in several phases:

analysis, design, development, and implementation.

Throughout all phases tests for assuring both

educational efficiency and usability are performed.

The process is iterated and the phases are repeated at

least partially until the training goals are met,

including the time and effort constraints.

3 USABILTY TESTING

In the framework of the WELKOM project the

usability is tested and assured by applying several

methods from different perspectives. On one hand,

tests are performed from the experts’ perspective –

in such a way we take advantage from the

accumulated knowledge in the domain. On the other

hand, tests are performed which give also the users’

perspective. In such a way the CAT system is

meeting the special requirements, abilities, learning

styles and expectations of the learners.

4 USABILITY TESTS DURING

THE DESIGN PHASE

In this phase the technical basis of the CAT system

is chosen and the initial design is being created. The

platforms and their options are compared with the

specific course requirements. The comparison is

based on the educational content and the trainees’

learning styles and preferences.

In order to explore the learners’ attitudes and

preferences, three questionnaires were used which

are presented in the following sections.

The first questionnaire is aimed at exploring the

trainees’ attitudes towards computers. Individuals’

attitude towards computers is a key component for

the understanding of the acceptance of, learning

success and satisfaction with CAT systems. The

questionnaire used was developed by M. Paprzycki

and D. Vidakovic (see Paprzycki et al., 1995). The

rationale behind its use is that different target groups

could have different attitudes ranging from fear and

confusion to pleasure and amusement, or the

computer might be perceived just as a tool for doing

the job. On the basis of the prevailing attitudes,

choices about the system design can be made. For

instance, the choices may be between a ‘formal’

CAT system or one with funny elements or even

using a game-like format.

The questionnaire for the attitude to computers

was administered during the development of CATS-

2. As expected, all of the trainees showed very

positive attitudes as all of them were using

computers at their work.

Recommendations concerning the CAT system

design were focused on functionality and

clarification of the actions that could be performed.

The second questionnaire used is David Kolb’s

Learning Style Inventory (LSI). This questionnaire

provides a framework for identifying students'

learning style preferences. The model postulates two

modes of getting experience – Concrete Experience

(CE) and Abstract Conceptualization (AC) – and

two modes of transforming experience – Reflective

Observation (RO) and Active Experimentation (AE)

(Kolb, 1984; Kolb, 2005; Kolb & Kolb, 2005). On

the basis of these two dimensions, four learning

styles can be characterized – converging (common

sense), dynamic, imaginative, and analytic.

Determining trainees’ learning styles helps in

designing the course format and especially how the

material should be presented – e.g. should the

material be presented in theoretical lectures or by

using case studies and exercises.

WEBIST 2007 - International Conference on Web Information Systems and Technologies

546

LSI was administered to the prospect trainees

who were supposed to use CATS-1, CATS-2, and

CATS-3. For CATS-2 the predominant modes of

learning are abstract conceptualization and active

experimentation (see Figure 1). Most of the trainees

belong to the converging type. They prefer technical

tasks, and are less concerned with people and

interpersonal aspects. People with converging style,

like to experiment with new ideas and to work with

practical applications. These learners like following

detailed sequential steps, hands-on activities, trial

and error, and being given clear objectives with a

logical sequence to activities. They learn through

interaction and thus computer-based learning is

more effective with them than other methods. Some

of the trainees belong to the analytic type. They like

abstract ideas and concepts, conceptual models,

designing experiments, reading, theories, and

structured activities. They enjoy a systematic

approach, detailed directions, and computer assisted

instructions.

Figure 1: LSI results involving 21 learners using CATS-2.

On the basis of these results the following

conclusions and recommendations concerning

CATS-2 were made. The training should focus on

thinking, not on feeling; clear, logical conceptual

schemes without details and examples should be

used; the trainees should have the opportunities to

solve problems and find solutions; more practical

asks should be included in the course.

The third questionnaire that was administered is the

VARK questionnaire assessing learner’s preferred

mode of learning and learners’ preferences for the

way they work with information. The four modes

considered are visual, auditory, reading, and

kinaesthetic. As a result of the test we can determine

the preferred mode of information processing of the

trainees. Knowing trainees’ learning styles

contribute to improve learning through improving

the information presentation mode. For example, if

most of the people prefer auditory presentation of

the information, such an option should be provided

in the systems (Fleming & Mills, 1992; Fleming,

2001).

The VARK questionnaire was administered for

CATS-1, CATS-2, and CATS-3. Results for CATS-

2 are presented here. The results show that most of

the learners’ possess a multimodal style. The

distribution is shown in Figure 2.

The main recommendations made were that

there should be a lot of practical exercises. The

material to be learned should be presented as a text.

Auditory presentation of the material is also possible

(however, this is not mandatory and if such mode is

provided, an option for switching off the auditory

presentation of the material should be easily

accessible).

R; 10

K; 14

V; 2

A; 7

Figure 2: VARK test results for 21 learners using CATS-2.

The total number is greater than 21 as some of the learners

have more than one strongly manifested style.

5 USABILITY TESTING DURING

THE DEVELOPMENT PHASE

In this phase the actual development of the CAT

system takes place.

In the IDM three different methods that provide

usability information for the development phase are

proposed: heuristic evaluation, user testing, and eye-

tracking recordings.

The heuristic evaluation of the system is

performed by trained experts. It is based on well-

defined and broadly accepted usability guidelines.

The aim of evaluation is to ensure that the CAT

system is built in a way that conforms to usability

standards and that information is presented in a

manner that maximizes its educational value As a

result of the evaluation, detailed recommendations

of the improvements of the system are provided

(more detailed description of the heuristic evaluation

method can be found in Nielsen, 1994).

The heuristic evaluation done during the

WELKOM project is made on the basis of 86 rules

divided in several categories. Heuristic evaluations

have been used for all three CAT systems. CATS-2

again will be presented here as an illustration. In the

beginning, two CAT prototypes were tested.

THE ROLE OF USABILITY TESTING IN AN INTEGRATED APPROACH TO CAT DEPLOYMENT - Experience

from the WELKOM project

547

Separate heuristic evaluations were performed with

both prototypes. Problems that were identified were

summarized in reports and presented to the

developers of the system. After consideration of the

severity of the problems on one hand and of the

technical possibilities on the other, one of the

prototypes was chosen. After that an additional

heuristic evaluation of the selected prototype was

performed and again the new recommendations were

implemented. This was done in several iterations.

The second method proposed is user testing

(Rubin, 2001; Lewis & Rieman, 1994; Nielsen &

Mack, 1994; Kunyavski, 2003; Dumas & Redish,

1999). In this type of study, representatives of the

trainees are asked to perform specified tasks with the

CAT system. Their actions and comments are

recorded and analyzed. It is important to test the

CAT system with real users, as neither the designers,

nor the usability experts can foresee all the problems

that the users could have in a task completion. The

difficulties experienced by the users are analyzed

and recommendations for the CAT system

improvement are given

User testing has been performed so far for

CATS-1 and CATS-3. For CATS-1, ten

representative tasks were selected (e.g., ‘Start the

system and log on’, ‘Start/stop/pause the lesson’,

‘Take a test’, and ‘Look at the test results’). The user

testing identified many additional problems in

comparison with the heuristic evaluation some of

which were crucial for the efficient work with the

CAT systems.

6 CONCLUSIONS

IDM has been applied so far up to the

implementation phase in which presently is only

CATS-1. This phase is actually the most interesting

and intense in terms of tests both for educational

efficiency and usability. A very important new

element during this phase will be the achievements

assessment which will be the basis for evaluating the

meeting of goal knowledge level.

Although no full iteration in IDM has been

performed so far, it can be said that combining

several methods during all the phases of CAT

deployment gives a lot of useful complementary

information that puts together the efforts of the

training stake-holders, the IDM team, the software

developers, and the educational content providers.

Thus IDM seems to allows not only the building of

an optimized CAT system but leads to economy of

time and efforts by making the prevention and the

solution of problems at the most appropriate

moments of the CAT development. Although the use

of so many tests might seem quite expensive the

authors believe that after the optimization of IDM at

the end of the project the final IDM methodology

will prove to be very efficient in terms of ROI. This

claim, however, is a research question which will be

dealt with in the remaining decisive one year of the

project.

ACKNOWLEDGEMENTS

All the work presented in this paper was supproted

by the WELKOM project. We would like to

acknowledge the fruitful discussions and

collaboration with the WELKOM partners and the

NBU team.

REFERENCES

Dumas, J. & Redish, J., (1999), A Practical Guide to

Usability Testing. Portland, OR: Intellect.

Fleming, N.D. & Mills, C. (1992). Not Another Inventory,

Rather a Catalyst for Reflection. To Improve the

Academy, 11, 137-155.

Fleming N. D. (2001) Teaching and Learning Styles:

VARK Strategies Honolulu Community College.

Kuniavsky, M. (2003). Observing the user experience: A

practitioner's guide to user research. Morgan

Kaufmann, San Francisco, CA.

Kolb, D. A. (1984). Experiential Learning: experience as

the source of learning and development. New Jersey,

Prentice Hall.

Kolb, D. A. (2005). The Learning Style Inventory Version

3.1. Boston, MA Hay Resources Direct.

Kolb, A. Y. & Kolb, D. A. (2005), The Kolb Learning

Style Inventory- Version 3.1. Technical Specifications.

Experience Based Learning Systems, Inc., Case

Western Reserve University.

Nielsen, J. & Mack, R.L. (Eds) (1994). Usability

Inspection Methods, . John Wiley & Sons, New York,

NY.

Lewis, C. & Rieman, J. (1994). Testing the design with

users. In: Task-Centered User Interface

Design.Retrieved November, 13, 2006 from

http://www.hcibib.org/tcuid/chap-5.html

Paprzycki, M., Vidakovic & Ubermanowicz, S. (1995).

Comparing Attitudes Toward Computers of Polish and

Americam Prospective Teachers. Technology and

Teacher Education Annual, AACE, Charlottesville,

VA, 45-48

Rubin, J. (2001) Handbook of Usability Testing: How to

Plan, Design, and Conduct Effective Tests. John Wiley

& Sons.

WEBIST 2007 - International Conference on Web Information Systems and Technologies

548