THE IMPORTANCE OF USABILITY CRITERIA ON LEARNING

MANAGEMENT SYSTEMS: LESSONS LEARNED

Aparecido Fabiano Pinatti de Carvalho and Junia Coutinho Anacleto

LIA - Advanced Interaction Laboratory, Department of Computing, Federal University of Sao Carlos, Brazil

Keywords: Usability, E-learning, Learning Management System, Learning Activity, Practical experience.

Abstract: This paper points to the importance of guaranteeing usability in Learning Management Systems (LMS) in

order to achieve success in performing learning tasks in this kind of environment. The paper presents the

results from a case study in which learners had to perform several learning tasks on the TIDIA-Ae LMS. It

is presented usability problems observed during the performance of the case study learning tasks, as well as

the difficulties that learners faced with in some tasks due to those problems and the actions taken in order to

make viable the execution of the learning activity tasks. By this paper, it is intended (i) to call LMS

developers’ attention to the importance of usability in the development of tools to support the learning

process; (ii) to point to interactive problems which should be avoided in LMS systems and (iii) to show

some problems which can appear during a learning activity execution supported by computer and possible

ways to deal with them.

1 INTRODUCTION

Nowadays, people live in a world where information

is very important and can define their professional

success. Professional life is also more complex and

competitive and, in order to keep updated and

guarantee their position, people need to acquire new

knowledge in a fast and effective way. For this

purpose, the advances in the computational area has

been used for supporting the learning process in an

approach that combines Electronic Learning (e-

learning) and Distance Learning (DL), so that

flexibility and portability can be provided to it.

Examples of using computational resources in the

learning process are the production and distribution

of learning content through the web.

In this context, several systems such as Moodle

(2008), Blackboard and WebCT (2008), and TIDIA-

Ae (2008), have been developed to manage the

distribution of learning content. These systems are

called LMSs (Learning Management System). This

paper points out that the usability of this kind of

systems is very important in order to reach success

in the learning process. It is necessary to consider

that, when the learning process happens via an e-

learning approach, learners not only have to learn a

new content but also have to learn how to use the

LMS functionalities to perform the proposed tasks.

So, the cognitive load demanded to learn the system

cannot be ignored because if the effort to understand

how to use the LMS itself is big, the learning

process success can be compromised (Carvalho et

al., 2008).

Usability can be defined as the set of all features

which allow users to interact satisfactorily with a

system in order to reach their goals with

effectiveness, efficiency and use satisfaction.

Effectiveness corresponds to the precision and

completion with which users reach their specific

goals; efficiency refers to the relation between

precision and completion of the reached goals and

the resources expended to achieve it; and the use

satisfaction is related to the comfort and acceptance

in using the system (Nielsen, 1993). It is noticed,

therefore, the importance of assuring the usability in

LMSs, because it is known that, the better the

system usability is, the easier for users is to

understand and to use it.

This paper discusses some usability problems

which were found in TIDIA-Ae LMS (TIDIA,

2008), during the performance of a learning activity,

i.e. a task that can promote the process of learning in

students, stimulating them to think and work on a

new concept or topic presented by a teacher. The

results presented here were not obtained from the

application of a usability evaluation method in the

system; the results were collected by the observation

of a practical experience. The paper is organized as

follow: section 2 describes the issues which concern

the system usability; section 3 presents a case study

where professors and undergraduate students have

performed a learning activity on the TIDIA-Ae

154

Fabiano Pinatti de Car valho A. and Coutinho Anacleto J. (2008).

THE IMPORTANCE OF USABILITY CRITERIA ON LEARNING MANAGEMENT SYSTEMS: LESSONS LEARNED.

In Proceedings of the Tenth International Conference on Enterprise Information Systems - HCI, pages 154-159

DOI: 10.5220/0001683901540159

 SciTePress

LMS, explaining the problems and pointing to the

solutions adopted in order to make viable the

learning activity tasks execution; finally, section 4

presents the conclusions about the reflection of the

usability problems in the learning activity execution

and point to some future work remarks.

2 USABILITY ISSUES

According to Nielsen (1993), there are ten main

issues which should be covered in a system so that a

good usability level can be reached. They are:

(1) learning facility; (2) use facility; (3) intuitive

interface; (4) simple and natural dialog; (5) feedback

to the user; (6) knowledge retention (remembering

facility); (7) high speed task performance; (8)

system prepared to avoid users’ errors; (9) consistent

error messages; and (10) subjective satisfaction.

A system with learning facility is the one which

is easy to learn. It means that the system offers

conditions for users to learn how to interact with it,

independently of the users’ level of knowledge and

skill, as well as to get the maximum performance

during the interaction (Dix et al., 2003; Nielsen,

1993). The interface predictability, synthesis

capacity, familiarity, generality and consistence are

some principles which support the learning facility

(Dix et al., 2003). The predictability

allows users to

preview the effects of their actions based on past

interactions with the system. The synthesis capacity

allows the users to know the effects of past actions

they performed so that they can go back to previous

steps and keep oriented in the interaction. The

familiarity

consists in the possibility of users to

apply the knowledge they get interacting with other

computer applications, in the current interaction.

The generality

is related to users interacting with

other computer applications in the same way they

are interacting with a specific system. Finally the

consistence

refers to the standardization of the input

and output mechanisms to reach similar goals.

The use facility is directly related to the system

flexibility. In order to present both learning facility

and use facility the system must be able to adapt

itself to the users’ knowledge and skill level

(Nielsen, 1993). Dix et al. (2003) defines flexibility

as the system ability to adapt itself to the context of

the interaction as well as to the users’ preferences

and needs, so that it can be used more efficiently.

The dialog initiation, the multi-threading capacity,

the switch facility among tasks and the system

adaptability are some important issues related to the

system flexibility. The dialog initiation

refers to the

interaction between the computer and the users

which should represent a partnership between the

two actors involved. It can be pre-emptory to the

system or pre-emptory to the users (Dix et al., 2003).

The multi-threading

is the system ability to keep a

plausible interaction with the user when more than a

task is happening at the same time. The switch

facility refers to the sharing of responsibility

between the user and the system, i.e. a specific task

being controlled either by the user or by the system.

This means that the user can set up the system to

perform a certain task or s/he can perform the task

manually (Dix et al., 2003). The system adaptability

concerns the changes in the user interface so that it

fits to the users’ level of knowledge (Nielsen, 1993).

The interface intuitivism is associated to the

system commands necessary to the tasks

performance and their memorization. The system

commands which are necessary to perform a specific

task should be clearly visible so that users do not

need to memorize them. However, it might be

pointed out the difference between the memorization

necessity and the memorization possibility (Nielsen,

1993). While the first forces the user to memorize

the system commands the second allows expert users

to raise their task execution speed.

In reference to the simple and natural dialog, it

is suggested using expressions and concepts which

are known by the users. For instance, technical

vocabulary from the computing area might be

avoided. Furthermore, irrelevant information might

be hidden from the users in order not to compete for

the user’s attention. According to Nielsen, it should

be presented only the necessary information to the

task performance (Nielsen, 1993).

The feedback to the user demands that the

interface provides mechanisms to notify the users

about the processing which is being performed and

the time which is going to take in order to be

concluded. It is necessary to provide mechanisms

which allow the users assess whether the system

behaviour is the expected and whether the

processing is still happening. These mechanisms are

extremely important for users to be always oriented

in the task execution.

About knowledge retention, the interface should

be designed so that users can remember how to

perform their tasks in the system after a period

without interacting with it. The user should be able

to remember the system main command without the

necessity to check instruction manuals. Moreover

the system help might be always visible so that the

users can use it when they need (Nielsen, 1993).

Concerning the task execution speed, the users

expect that the system answers them fast and with

precise information. The computer processing speed

and the algorithms to increase the system

THE IMPORTANCE OF USABILITY CRITERIA ON LEARNING MANAGEMENT SYSTEMS: LESSONS LEARNED

155

performance reflects directly on the system usability.

If the users have to wait lots of time to get their task

execution concluded, they get bored and the use

satisfaction is compromised. Consequently, the

system usability decreases (Dix et al., 2003).

The system preparation to avoid users’ errors is

related to the system robustness. The robustness is

one of the more important principles for maintaining

the system usability. It is related to the maintenance

given to the users so that the task execution is

successful. To be robust, a system should obey the

observance, the recovery, the comprehensibility and

the task conformity principles (Dix et al., 2003). The

observance

is related to the system feedback to the

user, i.e. mechanisms that allow users to observe and

assess the system behaviour, making users’

intervention and error correction possible. The

recovery

consists in the ability of users to take

corrective actions every time an error is recognized

(Nielsen, 1993). The comprehensibility

refers to the

way the users understand their communication with

the system (stability) and the conformity

refers to

the quality of the system support for users to

perform their tasks.

Although the system must be designed to

minimize users’ errors, it is impossible to predict all

different ways which users can interact with the

system and the errors that they can make. Thus it is

very important to design consistent error messages

that explain to users which action has caused the

error, where the error is located in the system and

how it can be corrected. In this way, the users are

always concerned about their actions and oriented to

solve the problems which appear during the

interaction, losing no time in correcting them.

Finally, the subjective satisfaction refers to how

users feel in using the system. It is directly related to

the system functionalities and the facility to use

them. If the system does not have the desired

functionalities the user will complain. In the same

way, the user will not be satisfied if the system has

all necessary tools for her/his task but they are very

difficult to be managed (Nielsen, 1993).

All these issues might be considered during the

system development in order to reach its

acceptability. This is because the users look for tools

that fit to their necessities, i.e. which support the

execution of their tasks and that do not bring

problems during their execution. Low usability

reflects on the system usefulness negatively (Dix et

al., 2003; Nielsen, 1993). Therefore, using a system

with low usability in e-learning can compromise the

execution of the learning tasks and, instead of

supporting the learning process, it can make it

unsuccessful, as it is discussed in the following.

3 LEARNING ACTIVITY ON

TIDIA-AE LMS – A CASE

STUDY

The FAPESP TIDIA-Ae Project (2008) aims at

research and development in DL supported by high

speed networks. The project goals include the

specification, design and implementation of several

DL tools based on flexible and cheap solutions. The

project LMS is currently under development. In its

first phase, it was developed a set of DL tools for the

LMS and it was played several learning activities on

it in order to perform a proof of concept (TIDIA,

2008). One of the learning activities played on the

environment was the learning activity “Guaranteeing

the continuity of home caring a sick person”, which

was planned by professors from the Nursing

Department of the Federal University of São Carlos

in the context of a case study whose main goal was

to illustrate the possible usages for common sense

knowledge to support learning activity planning

(Carvalho et al., 2007; Carvalho et al., 2008).

In the learning activity planning it was used the

OMCS-Br (Brazilian Open Mind Common Sense)

project knowledgebase (Anacleto et al., 2008). The

common sense knowledge was used to call the

learners’ attention to the way which the population

talks about requirements to be a caregiver or about

procedures which might be taken while home caring

a sick person. It was presented points to the learners

which they should emphasize during the orientation

and the community common vocabulary which

should be used by the future professionals in the

orientation process.

The learning activity was planned according to a

framework to prepare DL learning activities

supported by computers (Neris et al., 2007) and, as it

is defined in the framework, each learning activity

task was related to a computational tool, which

would be used to perform it. In the same way, as it is

proposed in the framework, each tool which would

be used for performing the activities was tested

before the learning activity starting. In the test, the

LMS tools could be used without any problems.

Nonetheless, the user who tested the tools, the

learning activity monitor, had previous knowledge in

computing and it is possible that his computing

background knowledge helped him not to feel

difficulties in using the system, differently from the

nursing learners as it is reported in the following.

It is worth mentioning that the results presented

in this paper are not from a usability evaluation such

as heuristic evaluation or user tests (Nielsen, 1993).

The results were empirically obtained in the case

study where learners had basic background in

ICEIS 2008 - International Conference on Enterprise Information Systems

156

informatics, i.e. were accustomed to use just

computer office tools such as text and spreadsheet

editors and web tools such as chats, forums and e-

mail clients. Table 1 lists the tools which presented

interaction problems during the first learning activity

tasks.

Table 1: TIDIA-Ae LMS tools not used in the learning

activity execution because of usability problems.

TIDIA-Ae LMS Tool Usability issues not attended

Chat

1. simple and natural dialog

2. feedback to the user

3. system prepared to avoid users’ error

4. consistent error message

5. users’ satisfaction

Hypertext

1. use facility

2. intuitive interface

3. simple and natural dialog

4. users’ satisfaction

Portfolio

1. learning facility

2. intuitive interface

3. users’ satisfaction

The first learning activity task was to explore the

TIDIA-Ae LMS tools which were going to be used

during the learning activity. As this activity was

guided by a monitor, it was decided to use the tool

Chat available in the LMS to conduct the task whose

purpose was to discuss the tools features and

possible difficulties to use them. Although the task

was quite simple, it could not be performed properly

because several learners had problems to open the

Chat. The problem was occasioned due the LMS

used an applet without digital signature to make the

tool available to users. This made a warning security

message to be presented to the learners, who did not

take the expected action to launch the tool. Figure 1

presents the warning security message presented to

the learners when they tried to initialize the TIDIA-

Ae Chat tool.

Figure 1: Warning security message presented when the

tool Chat is launched.

In order to launch the tool the learners should

have clicked on the button “Run”. However, some of

them thought that the message box was a pop-up and

clicked on the button “Close” (X) or in the button

Cancel when the box was presented. In this point it

can be considered that the usability issues simple

and natural dialog and system prepared to avoid

users’ errors was not attended. Concerning the first

issue previously mentioned, the message was

presented in English which was not the learners’

native language and it was used terms such as

“digital signature” and “ChatApplet” which were not

part of the learners’ vocabulary, according to what

could be verified by the analysis of the questions

which were sent by the learners to the learning

activity monitor.

Besides the learners who closed the digital

signature warning message box, others could not

launch the tool because of protection systems which

were installed in their computers and which did not

allow the warning message box to be presented to

the learners. This situation showed also the absence

of the issue feedback to the user, since it was not

presented any message warning the learners that the

message box was blocked.

In relation to the feedback to the user, another

system behaviour points that it does not works

properly. Figure 2 shows the interface that is

presented to learners, after the warning message box

previously mentioned being shown, even if the

learner clicks on the button “Close” or “Cancel” or if

the box is not presented to her/him due to pop-up

blockers.

Figure 2: Incorrect feedback to the users.

The message inside the red box says “Please,

wait while the connection is established with the

server…”. This message is not true if the user does

not click in the warning message box button “Run”,

since the applet is the responsible for trying the

connection. Hence, if the applet is not run, there is

no attempt to contact the server in order to start the

communication among the chat participants. In this

case, it should be presented a message notifying the

user that the connection could not be established

because the applet was not activated and explaining

to the user what s/he should do in order to get the

tool running properly. Thus, it can be considered

THE IMPORTANCE OF USABILITY CRITERIA ON LEARNING MANAGEMENT SYSTEMS: LESSONS LEARNED

157

that the system also does not provide consistent

error message to the user.

Because of the wrong feedback message

previously described some learners who were

performing the first learning activity task and who

had not clicked on the button “Run” kept waiting

more than five minutes for system response. As

some of them were in contact with each other, they

asked to one of the learners who had successfully

entered in the TIDIA-Ae Chat to notify the monitor

of the problems.

As more than 70% of the learners were

experiencing problems to enter into the Chat, the

monitor tried to conduct a chat section using an

instant messenger external to the LMS in order to try

to solve their problems. The section was not

successful because the learners were so disappointed

with the problems they were facing and full of

doubts that they asked lots of question at the same

time in such a way that the monitor could not answer

all questions in a suitable time. This was one of the

main lessons learned in the case study: do not try to

solve a general problem by trying to discuss it

distantly with everyone who is experiencing it at the

same time. It is very difficult to control everyone’s

anxiety so that a good communication can be

reached. The solution proposed by the monitor was

to send a tutorial explaining how to use each tool

which was going to be adopted during the learning

activity tasks so that the learners could perform

some sample tasks by themselves in order to learn

how to use the tools and report their difficulties by

e-mail so that the monitor could better help them. As

a consequence of this experience, the majority of

learners complained about the tool, demonstrating

that the tool did not provide subjective satisfaction.

By executing the tutorial activities, the learners

found problems in other tools, which were reported

by e-mail. After that, the Chat and two other tools of

the TIDIA-Ae LMS, the Hypertext and the Portfolio

mentioned in Table 1, were substituted by other

tools that learners felt comfortable to interact with.

This decision was taken because the problems in

those tools were impeding some learners to perform

the learning tasks and compromising the overall

performance of the learning activity.

Some problems with the tool Hypertext reported

by the learners pointed to the absence of the

usability issues use facility, intuitive interface,

simple and natural dialog and users’ satisfaction.

About the use facility, the tool Hypertext was

developed using some technologies which do not

Because of that, several learners could not perform

simple tasks using the tool. One of the problems

reported by the learners who were using Internet

the page edition area. Without this tool bar it was

possible only to add text to a page. It was not

possible to format the document, to add tables, to

add pictures, to add hyperlinks and so on, i.e. it was

not possible to create a real hypertext. The tool does

not offer the flexibility of being used properly in

different browsers, which makes the learner

unsatisfied with it. Furthermore several students

contacted the learning activity monitor because they

had not understood how to perform some tasks,

pointing to the use of specific terms from the

computational area; the interface did not provide a

simple and natural dialog. Learners have also

pointed to the tool unconformity with the way the

learners considered natural to interact with it, i.e. the

tool interface was not intuitive.

It was tried to persuade the learners to use the

work properly. However, some learners were

resistant to install and to use it. Some learners

justified that they were using computers in the

university department where Firefox was not

installed and where they had access to the computer

through a limited account, which did not allow them

to install any software. Others justified that they

were used to using Internet Explorer and they would

like to continue using it because they already knew

where they could find what they needed. This

showed the importance of a system to be prepared to

run in different platforms and to fit to the users’

needs. If the users do not feel comfortable with a

system or if the system does not provide the features

they expected, they will look for another one. This

was another lesson which could be learned in the

case study. As the tool Hypertext was discarded, the

learning tasks which supposed the collaborative

construction of documents were modified to

activities in which the learners should discuss their

ideas in specific forums, reach an agreement on

them and choose a representative who was

responsible for synthesize the group’s idea.

The tool Portfolio has been also criticized

because of its folder structures and the difficulties

which the learners faced to upload and download

documents. In this way, it was decided to use the

tool E-mail, which provided a suitable interaction, to

exchange information and documents of learners’

interest. The tool available to explore the disposed

learning content was also criticized because it did

not have a navigation tool bar. Thus, while exploring

a hyper document, the learners were not able to go

back to a previous page if there was no link “back”

in the page which they were exploring. However, as

the users could interact with the learning contend

ICEIS 2008 - International Conference on Enterprise Information Systems

158

reasonably, the tool was not discarded in the

learning activity.

4 CONCLUSIONS AND FUTURE

WORKS

This paper presented and discussed various usability

problems observed in TIDIA-Ae LMS tools during

the performance of a case study using that LMS. It

was shown that the absence of usability issues in an

application can lead users to give up using it and

adopting another application which provides a

consistent and satisfactory interaction. It was also

presented the actions taken during the case study

execution in order to make viable the learning

activity task execution.

Although this paper focused on a specific

experience, it presented real situations which can

occur in LMSs which lack the same usability issues

and reflect negatively on the learning activity

performance. It was shown the importance of

usability in e-learning systems, since an activity

could be not performed because of interactive

problems. As it was mentioned before, if the

usability of the system used to support the learning

process is low, the learner will have to expend a

bigger cognitive load to learn how to interact with it

and, consequently, will have less time to dedicate to

perform the learning tasks, which can compromise

the learning process. However, as it could be noticed

in this paper and according to Ardito et al. (2004),

some developers still put usability in second plan.

By this paper, it is expected to call e-learning

system developers’ attention to the fact that usability

must be approached since the beginning of the

system development in order to build a useful

product which can really support the learning

process. In addition to that, it is pointed some

usability problems identified in TIDIA-Ae LMS so

that developers can design their products in order

not to present the same problems. Furthermore, by

reporting solutions adopted to make viable the

learning activities task execution, such as to suspend

a task, to provide a tutorial with the purpose of

teaching the learner to interact with the LMS tools or

to choose other tools to execute a task, it is intended

to provide guidelines to people who are intending to

execute a distance learning activity supported by

computers so that they can deal with possible

problems with which they will face.

As the usability issues pointed in the TIDIA-Ae

LMS was empirically identified during the execution

of the case study, it is proposed, as future work, the

performance of usability tests involving all the

TIDIA-Ae LMS tools as well as the tools of other

LMSs mentioned in this paper, e.g. Moodle (2008),

Blackboard and WebCT (2008). Performed the tests,

it is intended to conduct a comparative study so that

the main usability problems in LMS tools can be

identified, classified and registered in a document

which can be used as reference by developers of this

kind of systems.

ACKNOWLEDGEMENTS

We thank Fapesp and CAPES for financial support,

Prof. Silvia H. Zem-Mascarenhas and Prof. Rosely M.

de Figueiredo from DEnf/UFSCar, and the students

who participated on the learning activity.

REFERENCES

Anacleto, J. C. ; Carvalho, A. F. P. DE ; Pereira, E. N. ;

Ferreira, A. M. ; Carlos, A. J. F. . Machines with good

sense: How can computers become capable of sensible

reasoning? In: WCC 2008, Milan. Proceedings... /In

press/

Ardito C.; De Marsico, M.; Lanzilotti R.; Levialdi, S.;

Roselli, T.; Rossano, V., Tersigni, M. Usability of e-

learning tools. In: Working Conference on Advanced

Visual Interfaces, 2004, Gallipoli (Italy).

Proceedings… ACM Press, 2004. p. 80-84.

Blackboard & Webct. Available on:

http://www.webct.com/. Access in: March, 2008.

Carvalho, A. F. P. DE; Anacleto, J. C.; Zem-Mascarenhas,

S. Planning Learning Activities Pedagogically

Suitable by Using Common Sense Knowledge. In:

International Conference on Computing (CIC 2007),

2007, Proceedings. p. 1-6.

Carvalho, A. F. P. De; Anacleto, J. C.; ZEM-

MASCARENHAS, S. Learning Activities on Health

Care Supported by Common Sense Knowledge. In:

ACM SAC, 2008, Fortaleza. Proceedings. ACM Press,

2008. p. 1-5.

Dix, A. J., Finlay, J. E., Abowd, G. D., Beale, R. Human

Computer Interaction, 3

ed., 2003.

Neris, V. P.; Anacleto, J.; Zem-Mascarenhas, S.;

Carvalho, A. F. P. PACO - A Framework for Planning

Learning Activities Supported by Computers. In: 18

Brazilian Symposium on Informatics in Education

(SBIE 2007), 2007, São Paulo. Anais... p. 597-606.

nielsen, J. Usability Engineering. Morgan Kaufmann

Publisher, Inc. San Francisco California, 1993

MOODLE – A free, open source, course management

system for online learning. Available on:

http://moodle.org/. Access in: March, 2008.

TIDIA – Electronic Learning. Available on: http://tidia-

ae.incubadora.fapesp.br/portal. Access in: March,

2008.

THE IMPORTANCE OF USABILITY CRITERIA ON LEARNING MANAGEMENT SYSTEMS: LESSONS LEARNED

159