What Makes an LMS Effective
A Synthesis of Current Literature
Nastaran Zanjani, Shaun Nykvist and Shlomo Geva
Queensland University of Technology, Brisbane, Australia
Keywords: Learning Management Systems (LMS), e-Learning, Higher Education, e-Learning Evaluation Framework.
Abstract: There is a growing number of organizations and universities now utilising e-learning practices in their
teaching and learning programs. These systems have allowed for knowledge sharing and provide
opportunities for users to have access to learning materials regardless of time and place. However, while the
uptake of these systems is quite high, there is little research into the effectiveness of such systems,
particularly in higher education. This paper investigates the methods that are used to study the effectiveness
of e-learning systems and the factors that are critical for the success of a learning management system
(LMS). Five major success categories are identified in this study and explained in depth. These are the
teacher, student, LMS design, learning materials and external support.
1 INTRODUCTION
Providing an environment for e-learning (electronic
learning) and more recently the notion of blended
learning in the Higher Education sector has
increased the need for these institutions to invest in
learning management systems (LMS) to support
teaching and learning. This can be viewed as an
attempt to be more competitive by attracting a larger
market share of students (Turner and Stylianou,
2004). At first, e-learning systems were mostly used
to provide external and distance education
modalities and to offer more flexibility for external
students (Mason, 2006; Allen and Seaman, 2007).
However, e-learning has been considered an
essential part of the teaching and learning process
for many Higher Education Institutions (HEI) and
many of these institutions are further adopting a
combination of face-to-face (F2F) learning and e-
learning (Keengwe and Kidd, 2010; Palloff and
Pratt, 2001), which is referred to as blended
learning. Rapidly growing developments in
information and communications technologies (ICT)
provide an opportunity for students to learn in a
more flexible environment where access to learning
materials and resources is available at all times
regardless of the time and place.
However, many users stop using virtual learning
tools such as those embedded within an LMS after
first practices (Sun et al., 2008). The continued use
of the available tools is not high (Chiu et al., 2007)
and the discontinuation of e-learning tools after the
first adoption, occurs frequently. While initial
adoption is an important success factor of using an
e-learning system, continuous use is still required to
achieve tangible success. Initial e-learning adoption
and continued use of LMS tools are two significant
concerns in this area.
In an attempt to further explore the LMS critical
success factors, this paper reviews and synthesizes
different approaches that have been used to study the
effectiveness of e-learning systems and discusses the
significant factors that help an LMS to be used most
efficiently.
2 APPLYING A FRAMEWORK
In analysing and understanding the effectiveness of a
particular system frameworks or models are often
developed and applied to a body of work. It is within
this context that computer human interaction, self
regulated learning, collaborative learning, and e-
learning acceptance studies, can aid in a deeper
understanding of the critical issues that entice users
to actively contribute within virtual learning
environments. To study the behaviour of LMS users,
an overview of seven of the most relevant studies is
presented in this section.
574
Zanjani N., Nykvist S. and Geva S..
What Makes an LMS Effective - A Synthesis of Current Literature.
DOI: 10.5220/0004384905740579
In Proceedings of the 5th International Conference on Computer Supported Education (CSEDU-2013), pages 574-579
ISBN: 978-989-8565-53-2
Copyright
c
 2013 SCITEPRESS (Science and Technology Publications, Lda.)

2.1 Technology Acceptance Model
The technology acceptance model (TAM) is a
framework that has been applied in many technology
adoption studies. Its focus is to predict and assess
user willingness to accept technology. TAM
investigates the relationships between perceived ease
of use (PEU), perceived usefulness (PU), and
attitudes and intention in adoption. User perception
and attitudes toward the system is influenced by
both these factors.
This framework can be used as a tool to predict
learning satisfaction in virtual learning environments
and show the perceived ease of use and perceived
usefulness of the LMS that significantly affects
student satisfaction (Pituch and Lee, 2006). The
application of TAM in relation to LMS’s identifies
the perceived ease of use in student perception of
using an e-learning system and the perceived
usefulness in the perception of the learning
enhancement level as a result of the adoption of the
system. Perceived usefulness has more impact on
intention to e-learning adoption than perceived ease
of use which means if students find e-learning
objects difficult to use they will rate them as less
useful.
2.2 Unified Theory of Acceptance
and use of Technology
Venkatesh and Davis (2000) extended the TAM
framework and introduced cognitive instrumental
processes and social influence as effective factors on
perceived usefulness and usage intention and as a
result introduced the Unified Theory of Acceptance
and Use of Technology (UTAUT) model (Venkatesh
and Davies, 2000). This framework was based on
the findings of eight models in Information System
(IS) adoption area. It hypothesized three factors that
directly predict the intention to use an invention
which are:
 Effort expectancy or perceived ease of use
 Performance expectancy or short term perceived
usefulness and
 Social influence
It also argues that intention and facilitating
conditions were direct predictors of usage behaviour.
2.3 Perceptions of Innovation
Characteristics
The Perceptions of Innovation Characteristics (PCI)
is yet another model that can be applied in e-learning
acceptance studies. This model explains the key
innovation attributes that impact on the user
acceptance. Critical innovation attributes identified
in this model are (Rogers, 1995):
 Compatibility which refers to the degree a system
is consistent with the learner current requirements,
values and previous experiences
 Trialability which specifies the user perception of
a chance to try the system prior to using it on
commitment
 Relative advantage which is the level of
enhancement a system offers in comparison to
previous tools for accomplishing the same task
 Complexity which is the level of perceived
complexity to learn and use the system
 Observability which shows the result visibility of
adopting the system
Moore and Benbasat (1991) extended the model to
seven constructs including:
 Compatibility
 Trialability
 Relative advantage
 Ease of use
 Result demonstrability which explains how much
the outcomes of using the system are tangible
 Visibility which is defined as “the extent to which
potential adaptors see the innovation as being
visible in the adoption context” and finally
 Image which is the feeling that using a system can
enhance user social status
2.4 Expectation Confirmation Model
Expectation Confirmation Model (ECM) is a
framework that has been applied to study the
intention to continue using innovations. This model
defines five constructs to explain the consumers
level of satisfaction (Oliver, 1980). These steps are:
1. Customers form a preliminary expectation of a
particular service or product before purchasing it.
2. Then they accept and make use of that service or
product.
3. After a period of first consumption, users form
conceptions about the service or product efficiency.
4. Then, consumers determine the level of
confirmation of expectations by comparing their
perceptions of the service or product performance
with their initial expectation. Accordingly, based on
the confirmation level, a dissatisfaction or
satisfaction feeling is formed.
5. Finally, satisfied users decide to continue using
the service or product in the future, whereas
dissatisfied consumers stop its later use.
WhatMakesanLMSEffective-ASynthesisofCurrentLiterature
575

Marketing studies have indicated that consumer
level of satisfaction is the main reason to repurchase
intention (Szymanski and Henard, 2001).
Bhattacherjee suggested to apply ECM in IT
acceptance and continued use intention because of
the similarity between the user intention to keep
using IT products and the consumer intention to
repurchase a service or product (Bhattacherjee,
2001). This model hypothesizes that a user decision
to continue using Information technology is affected
by three factors:
 Satisfaction
 Expectations confirmation and
 Post-acceptance expectations
2.5 Flow Experience Theory
Flow experience theory is a tool that can be used to
explain the e-learning adoption. This theory explains
the holistic experience that individuals perceive
when they are totally engaged in performing a task
(Csikszentmihalyi, 1997). In this case, people
become so involved in their ongoing activities that
they are unable to identify any variation in their
surroundings. From the perspective
of motivating people to use information technology,
Flow experience can be considered as an intrinsic
motivation construct in comparison with perceived
usefulness, which reflects user extrinsic incentive
(Lee, 2010). Its different constructs such as:
 Concentration and focus (Lee, 2010), (Li and
Browne, 2006)
 Enjoyment (Lee, 2010),
 Attention
 Control
 Curiosity
are variables that have been used to measure Flow
experience.
2.6 Theory of Planned Behaviour
The Theory of Planned Behaviour (TPB) is also a
framework that can be applied in e-learning
acceptance studies. Based on this model, the factors
that influence behavioural intention are (Ajzen,
1991):
 Perceived behavioural control which refers to the
availability of required skills and resources as well
as the user perception of the necessity of getting the
results.
 Subjective norm which reflects the perceived social
demands to show the behaviour and is related to
social normative beliefs about the expectation and
 Behavioural attitude which refers to how favourite
a person evaluates the behaviour
A number of studies have examined these three
variables and found them valid to explain a persons
decision to use IT (Liao et al., 1999). Applying TPB
to the e-learning field, perceived behavioural control
explains the role of users basic internet skills and the
subjective norm argues the impact of peers attitude
on a student to uptake using an LMS (Lee, 2010).
2.7 Media Richness Theory
The last applicable framework, Media Richness
Theory (MRT) aims to explain the characteristics of
technologies that decrease hesitancy and ambiguity
in different business settings and can reveal another
aspect of appropriate e-learning systems.
Researchers showed that the media with more
communication channels was more efficient for
presenting materials that require less analysis.
However, to present materials that need more
analysis, lean media, like text, was more effective
than rich media, like video. The concepts of
selecting appropriate media has also been
investigated in education (Liu et al., 2009).
3 E-LEARNING ADOPTION
AND CONTINUED
USE FACTORS
Researchers (Sun et al., 2008) have adopted these
models individually or as a combination, to
investigate success factors of e-learning acceptance
and continued use. Five main critical success factors
can be synthesised from the findings. These are
identified as teacher attitude and skills, student
attitude and skills, LMS design, Learning materials
characteristics and external support. This section
explains each factor in further detail.
3.1 Teacher Attitude and Skills
Instructor behaviour toward virtual learning
environments (Sun et al., 2008) as well as the
technical capabilities (Benson Soong et al., 2001)
has an important role in student behaviour towards
an LMS system. In a survey conducted on 900
students, Selim (2007) showed that the instructor’s
interactive learning attitude is one of the most
critical factors that can entice students to actively
interact on an e-learning system.
CSEDU2013-5thInternationalConferenceonComputerSupportedEducation
576

3.2 Student Attitude and Skills
The computer knowledge and previous experiences
of students are also significant factors that influence
adoption and acceptance of an e-learning system.
Researchers have identified the self efficiency and
experience of students using the technology as a
significant factor (Chiu and Wang, 2008) while
anxiety can be identified through a lack of
knowledge and skill and can cause a major negative
outcome (Sun et al., 2008).
3.3 LMS Design
One of the most important factors that impacts on
using an LMS is how the system is designed.
Different characteristics have been highlighted by
researchers that can affect the intention to use an e-
learning system which are:
 Perceived ease of use which is sometimes called
effort expectancy (Chiu and Wang, 2008) or
perceived behavioural control (Lee, 2010).
 The interactivity of the system (Pituch and Lee,
2006)
 User friendly interface design (Selim, 2007)
 Complete and easy to understand information of
the system
3.4 Learning Materials Characteristics
The learning content that is delivered through an
LMS system and the pedagogical approaches that
are followed in a unit should also support e-learning.
Effective learning content should consider the
following criteria:
 Perceived usefulness
 Accuracy and completeness
 Performance expectancy which is referred to the
level of short term perceived usefulness
 Intrinsic value which is referred to the degree an
activity is delightful
 Utility value which is the relevance of a system to
future career objectives or long term usefulness
 The collaboration level of learning materials
 The compatibility between learning objects and
student requirements
 Relative advantage
3.5 External Support
Support from the institute and the peer impact are
also important factors to be considered in the use
and implementation of an LMS. The educational
organization should provide:
 Computer labs
 Reliable networks
 Technical troubleshooting
 Information accessibility
Subjective norms or the peer impact also proved to
motivate individual intention to continue using an e-
learning system (Lee, 2010).
4 DISCUSSION
A synthesis of the literature and frameworks
presented here allows for the identification of a
three-phase timeline explaining student behaviour.
Although the important components in each phase
overlap the factors in the other phase, this
classification gives a better understanding of the
required steps that lead to the effective
implementation and use of an LMS (see figure 1).
The timeline and the relevant critical factors in each
phase are discussed in this section with more details.
The first phase deals with the essential factors that
need to be considered before the use of the LMS.
Previous student computer skills and knowledge of
ICT is one of the most important pre-requisites of a
successful LMS experience (Chiu and Wang, 2008).
The more a student is ICT competent, the less effort
that is required for the student to adopt and use the
e-learning system. However, more experienced users
may expect more advanced features in the system.
The second phase of an LMS acceptance occurs
while students start using the system. In this phase,
the LMS design, the quality of learning materials,
teacher attitude and external support plays a
significant role in encouraging students to continue
using the e-learning system. In terms of LMS
design, the richness of the media used in presenting
the learning martials (Liu et al., 2009), the response
time, the interactivity of the system (Pituch and Lee,
2006,), a logical navigation structure, easy and good
access (Sun et al., 2008) and the design of the
interface all impact on the students intention to
continue participating in an e-learning environment.
The Learning materials must be compatible with
the virtual learning environment (Papp, 2000), have
a level of joyfulness (Chiu et al., 2005), be accurate
and comprehensive. Consequently, the design and
use of teaching experiences that enhance the sense
of equality and community among students can
encourage students to be more active in virtual
learning environments. While the design of the
WhatMakesanLMSEffective-ASynthesisofCurrentLiterature
577

teaching experiences is quite important the
assessment model used must also be supportive of
this. The assessment can play a large factor in
student attitude toward the use of the LMS ( Sun et
al., 2008)
In this phase, the behaviour of peers is also seen as
an external motivation and may change how a
student interacts with the LMS since social influence
is proved to be effective on an individuals attitude
(Lee, 2010). The availability of computer labs, the
network reliability, troubleshooting support and
information accessibility are other required external
supports that are vital in the success of an e-learning
program (Selim, 2007).
Considering the ECM model discussed earlier, the
users first experiences of the system forms their
ongoing approach and continued use which
is the third phase of the timeline (see figure 1).
Figure 1: LMS adoption and Acceptance Timeline.
The users short term and long term perceived
usefulness of the system (Lee, 2010), (Pituch and
Lee, 2006), and the feeling of relative advantage in
using an LMS (Liao and Lu, 2008), as well as the
tangibility of the results and the level of
confirmation of initial expectances create a level of
user satisfaction that forms continued use behaviour.
5 CONCLUSIONS
E-learning and more recently blended learning is
used significantly in higher education teaching and
learning, however, it is critically important to use it
in ways that support and promote positive learning
experiences for students. The existence of learning
tools in an LMS does not automatically result in
them being used for positive or effective teaching
and learning experiences. The synthesis of literature
presented here highlights the critical factors that
affect the success of an LMS and how to plan for
success. The five factors involving teachers,
students, learning materials and pedagogical
approaches, LMS features and design, and external
supports need to form an essential component of a
successful LMS implementation. An LMS
implementation needs to be planned and teaching
staff need to be aware of the factors that can lead to
its successful use as an essential component of
teaching and learning within higher education.
REFERENCES
Ajzen, I. 1991. The theory of planned behavior.
Organizational behavior and human decision
processes, 50, 179-211.
Allen, E. & Seaman, J. 2007. Online Nation: Five Years of
Growth in Online learning. .The Sloan Consortium:
Needham, MA. .
Benson Soong, M., Chuan Chan, H., Chai Chua, B. &
Fong Loh, K. 2001. Critical success factors for on-line
course resources. Computers & Education, 36, 101-
120.
Bhattacherjee, A. 2001. An empirical analysis of the
antecedents of electronic commerce service
continuance. Decision support systems, 32, 201-214.
Chiu, C. M., Sun, S. Y., Sun, P. C. & Ju, T. L. 2007. An
empirical analysis of the antecedents of web-based
learning continuance. Computers & Education, 49,
1224-1245.
Chiu, C. M. & Wang, E. T. G. 2008. Understanding Web-
based learning continuance intention: The role of
subjective task value. Information & Management, 45,
194-201.
Csikszentmihalyi, M. 1997. Finding flow: The psychology
of engagement with everyday life, Basic Books.
Govindasamy, T. 2001. Successful implementation of e-
learning: Pedagogical considerations. The Internet and
Higher Education, 4, 287-299.
Keengwe, J. & Kidd, T. T. 2010. Towards Best practices
in Online Learning and Teaching in Higher Education.
MERLOT Journal of ONLINE Learning and Teaching
in Higher Education 6, 533-541.
Lee, M. C. 2010. Explaining and predicting users’
continuance intention toward e-learning: An extension
 Media richness
 Response time
 System interactivity
 Easy navigation and access
 Interface design
LMS
Design
 Compatibility with virtual
environment
 Joyfulness
 Accuracy
 Fulfilment
 comprehensibility
 Teaching techniques that
enhance the sense o
f
equality and community
Learning
materials
experiences
• Peers impacts
• Computer labs
• Network reliability
• Troubleshooting support
• Information accessibility
External
support
• Short term and Long term
perceived usefulness
• Relative advantage
• Result demonstrability
• Confirmation
Student
 Trial ability
 Compatibility
LMS
Design
Computer Skills
and
IT
Knowledge
Teacher
&
Students
While
using
an
LMS
Before using
an
LMS
After using an
LMS
CSEDU2013-5thInternationalConferenceonComputerSupportedEducation
578

of the expectation–confirmation model. Computers &
Education, 54, 506-516.
Leidner, D. E. & Jarvenpaa, S. L. 1993. The information
age confronts education: Case studies on electronic
classrooms. Information Systems Research, 4, 24-54.
Li, D. & Browne, G. J. 2006. The role of need for
cognition and mood in online flow experience.
Journal of Computer Information Systems, 46, 11-17.
Liao, H. L. & Lu, H. P. 2008. The role of experience and
innovation characteristics in the adoption and
continued use of e-learning websites. Computers &
Education, 51, 1405-1416.
Liao, S., Shao, Y. P., Wang, H. & Chen, A. 1999. The
adoption of virtual banking: an empirical study.
International Journal of Information Management, 19,
63-74.
Liaw, S. S. 2008. Investigating students’ perceived
satisfaction, behavioral intention, and effectiveness of
e-learning: A case study of the Blackboard system.
Computers & Education, 51, 864-873.
Liu, S. H., Liao, H. L. & Pratt, J. A. 2009. Impact of
media richness and flow on e-learning technology
acceptance. Computers & Education, 52, 599-607.
Mason, R. 2006. The university – current challenges and
opportunities. D'Antoni, S.(2006), The Virtual
University. Models and Messages, UNESCO-IIEP.
Moore, G. C. & Benbasat, I. 1991. Development of an
instrument to measure the perceptions of adopting an
information technology innovation. Information
Systems Research, 2, 192-222.
Oliver, R. L. 1980. A cognitive model for the antecedents
and consequences of satisfaction. Journal of
Marketing Research, 17, 460-469.
Palloff, R. M. & Pratt, K. 2001. Lessons from the
Cyberspace Classroom: The Realities of Online
Teaching. San Francisco: Jossey-Bass.
Papp, R. 2000. Critical success factors for distance
learning. Americas Conference on Information
Systems, Long Beach, CA, USA.
Pituch, K. A. & Lee, Y. 2006. The influence of system
characteristics on e-learning use. Computers &
Education, 47, 222-244.
Selim, H. M. 2007. Critical success factors for e-learning
acceptance: Confirmatory factor models. Computers &
Education, 49, 396-413.
Sun, P. C., Tsai, R. J., Finger, G., Chen, Y. Y. & Yeh, D.
2008. What drives a successful e-Learning? An
empirical investigation of the critical factors
influencing learner satisfaction. Computers &
Education, 50, 1183-1202.
Szymanski, D. M. & Henard, D. H. 2001. Customer
satisfaction: a meta-analysis of the empirical evidence.
Journal of the academy of marketing science, 29, 16-
35.
Turner, W. L. & Stylianou, A. C. 2004. The IT advantage
assessment model: Applying an expanded value chain
model to academia. Computers & Education, 43, 249-
272.
Venkatesh, V., Davis. F.D 2000. A theoretical extension
of the technology acceptance model: four longitudinal
field studies. Management Science, 46, 186-204.
Wang, Q. 2009. Design and evaluation of a collaborative
learning environment. Computers & Education, 53,
1138-1146.
WhatMakesanLMSEffective-ASynthesisofCurrentLiterature
579