ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED

QUALITY OF LMS SYSTEM AND E-COURSE DELIVERY

Matjaž Debevc, Martina Breg

Faculty of Electrical Engineering and Computer Science, University of Maribor, Smetanova ulica 17, Maribor, Slovenia

Julija Lapuh Bele

B2 d.o.o., Ljubljana, Slovenia

Keywords: Distance Learning, e-Learning, Learning Management Systems, e-Course, Learning Evaluation.

Abstract: In this research, a quality of e-Learning has been measured on the basis of students’ satisfaction with e-

learning environment (i.e. LMS system) and e-Course delivery. A questionnaire has been developed to

measure perceived quality of e-Learning technology and didactics. Results of this research also show

statistically significant correlation between the quality of e-Learning environment and the e-Course quality

in case of blended learning mode of delivery where only short introductory meeting and final examination

are carried out face-to-face. Students who were satisfied with e-Learning environment were also satisfied

with e-Course and vice versa. The research has been conducted by taking into consideration two different

Learning Management Systems and eight e-Courses facilitated with different teachers.

1 INTRODUCTION

e-Learning can be described with three basic criteria:

learning on demand, transfer of information to

students, and implementation of virtual classroom

(Rosenberg, 2001).

The success of any learning program is largely

dependent on the motivation and attitude of learners.

Therefore the key factors in e-learning delivery are

usability and didactic effectiveness (Ardito,

Costabile, Marsico, Lanzilotti, Levialdi, Roselli &

Rossano, 2006).

Since ICT is a crucial factor in each e-learning

setting, researchers emphasise various aspects to

evaluate e-learning platforms and other learning

applications (Ardito et al, 2006, Costabile, Marsico,

Lanzilotti, Plantamura & Roselli, 2005, Dringus &

Cohen, 2005, Squires & Preece, 1999, Zaharias &

Poylymenakou, 2006). Quality of computer software

often means usability (Nielsen, 1994).

Unfortunately, there is a lack of an evaluation

model that enables quick and easy evaluation of

e-Learning quality aspects. Existing models and

questionnaires are time consuming and

comprehensive (Bates & Obexer, 2005, Philips,

2005, Nielsen, 1994, Kirakowski, 1993). Since

students do not like to fill out long questionnaires,

such questionnaires are thus inappropriate for

regular implementation after e-Courses.

Therefore, this paper focuses on the newly

designed short-time e-learning quality evaluation

model, which we call DEMA model. The

questionnaire consists of only 28 items that cover

the following indicators of students’ satisfaction

with LMS system and e-Course delivery:

 Usability,

 Users Communication,

 Functionality,

 Safety,

 Help and Support,

 Learning Satisfaction,

 e-Content Satisfaction,

 Gained Knowledge and

 Transferred Knowledge into Practise

The questionnaire is short enough to be

implemented after the completion of each e-Course.

In this study, the following research question is

also raised: is there a correlation between the

perceived e-learning portal quality and the e-Course

quality? Therefore, the following hypothesis has

311

Debevc M., Breg M. and Lapuh Bele J..

ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE DELIVERY.

DOI: 10.5220/0003304503110315

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 311-315

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

been set: there is statistically significant relationship

between the quality of e-Learning environment and

the quality of e-Course. In the study, the term

e-Learning environment refers to e-Learning portal,

There were total 197 students and 8 teachers

involved in the study. Each student has attended one

of eight blended learning courses (i.e. blend of

face to face and facilitated e-Learning) and has been

mentored by one of eight teachers. Face-to-face part

of each blended learning course consisted of a short

introductory meeting and final examination in

classic classroom. The major part of each course has

been conducted online. Participating students have

used learning environments eCampus (Debevc &

Bele, 2008) or Moodle (Martin-Blas & Serrano-

Fernandez, 2009).

2 DEMA

MODEL - QUESTIONNAIRE

In order to evaluate the quality of e-Courses and to

design our own questionnaire, we have chosen

Kirkpatrick’s model (Kirkpatrick & Kirkpatrick,

2006) and adjusted it to the population that

participated in the study.

Disadvantage of Kirkpatrick’s model is in its

complexity, in relation to a number of questions and

the time needed to conduct evaluation and data

analysis. Therefore, we have tried to search for a

shorter, faster model, which would confirm our

hypothesis that there is statistically an important

relationship between the quality of e-learning

environment (LMS system) and the quality of e-

Course.

To reach these requirements, the Kirkpatrick’s

model, based on his first three levels, needed to be

redesigned.

The goal of the anonymous DEMA

questionnaire is to measure learners’ satisfaction

with e-Learning fundamentals, e-Learning

environment and e-Course as well as to confirm or

reject the hypothesis that the quality of e-Learning

environment and e-Course quality are significantly

correlated.

2.1 First Part - Quality of e-Learning

Environment

The first part of the questionnaire evaluates the

quality of e-Learning environment. This part

consists of thirteen 1-to-5 rating Likert scales of

quality measures divided into five categories and is

based mainly on European Computer Driving

License Certified Training Professional trainers

evidence record (ECDL CTP, 2010), adapted for

DEMA model.

It measures simplicity of use, ability of

communication between users, functionality,

security and additional help and support.

2.2 Second Part - e-Course Quality

The second part of the questionnaire evaluates the e-

Course quality based on first three levels of

Kirkpatrick’s model. It consists of fifth teen 1-to-5

rating Likert scales of quality measures divided into

five categories. It measures learner’s satisfaction

with course organisation, given information,

teacher’s support and meeting individual

expectations. The second factor of e-Course quality

is the e-content quality. The questionnaire measures

intelligibility, conciseness, graphic design, quality of

used multimedia and hyperlinks and quality of tests

for knowledge evaluation.

The next factor measures the gained knowledge.

Even though in most cases, the gained knowledge is

evaluated with written tests, we have measured

participant’s opinion on gained knowledge.

The evaluation ends with questions about

usefulness of gained knowledge (i.e. transfer of

knowledge into practice).

3 RESEARCH

The purpose of this study is to confirm or reject the

following hypothesis: there is a positive relationship

between quality of e-Learning environment (i.e. e-

Learning portal powered by LMS system) and e-

Course quality in blended learning setting.

3.1 Methodology

3.1.1 Participants

Full-time students from Faculty of Electrical

Engineering and Computer Science (FERI) and part-

time students from B2 Vocational College have

participated in the study. Each student has attended

one of eight blended learning courses (i.e. blend of

face to face and facilitated e-Learning) and has been

mentored by one of eight teachers.

Students at FERI used LMS system Moodle

while those students at B2 learned via LMS system

eCampus.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

312

3.1.2 Procedure

All participants attended blended-learning courses

(BL). Each blended learning course started in the

classroom, with an introductory face to face (F2F)

meeting. Teacher presented the course, announced

learning goals, learning tasks (e.g. project work,

assessments), code of behaviour within the course

and gave advice on e-Learning strategies. Then, the

e-Learning activities began. Most of them run

asynchronously. Students learned from e-content

(i.e. interactive learning materials that included

automated feedback and dynamic graphics such as

video, animations and simulations) and performed

learning tasks according to a weekly schedule that

determines learning activities (i.e. real-time online

meetings, readings, discussions, project work, on-

line assessments etc.). All activities had deadlines.

Students could carry them out according to their

own schedule. Teacher used the following activities

to facilitate learning:

 Followed student work and monitored their

progress using LMS tools.

 Facilitated, motivated and encouraged students

using communication tools.

 Stimulated communication and collaboration

among students.

 Actively participated, promoted and led

interactive discussions.

 Provided answers to questions, feedback and

recommendations on course activities.

The exam was the last activity in the blended

learning course and the examination took place in

the classroom. After the exam, the students filled out

questionnaires.

The research was carried out during a period of

one semester.

3.1.3 Measurements

All participants filled out a questionnaire where they

specified gender, age group, school, course,

employment status and expressed their opinion on

the quality of e-Learning environment and e-Course

quality.

As we expected that individuals would attempt

to quantify constructs, which have not been directly

measurable, we have used multiple-item scales and

summated ratings to quantify the constructs of

interest (i.e. e-Learning environment quality, e-

Course quality). The quality of e-Learning

environment is measured with a scale of 13

questions and e-Course quality is measured with a

set of 15 questions.

For all questions, 1-to-5 rating Likert scale (1-

strongly disagree, 2-disagree, 3-neutral, 4-agree, 5-

strongly agree) is used. New variable portal quality

is calculated as the arithmetic mean of 13 values,

which are measured with questions about aspects of

e-Learning environment quality. The variable e-

Course quality is calculated as the arithmetic mean

of 15 values is measured with questions about

aspects of e-Course quality. Both variables are

numerical.

3.1.4 Statistical Treatment

Since a new questionnaire is developed, it has to be

found out whether the instrument is reliable.

Therefore, Cronbach's alpha is computed to measure

the reliability (internal consistency) of scales for e-

Learning environment quality and e-Course quality.

Descriptive statistics are used to analyse

demographics data.

Pearson's correlation is performed to determine

if there is a significant relationship between the e-

Learning portal quality and e-Course quality. A

significance level of p < 0.05 is adopted for the

study.

SPSS is used for data analysis.

3.2 Results and Interpretation

3.2.1 Reliability of the Questionnaire

For each scale (e-Learning environment quality and

e-Course quality), the Cronbach’s α is used to check

how closely a set of items is related as a group.

Values of Cronbachs’ coefficients α are 0.913

(for the scale e-Learning environment quality) and

0.937 (for the scale e-Course quality). Since each

value is greater than 0.8, it can be concluded that the

questionnaire is sufficiently reliable. Alpha

coefficients, for both scales, are above 0.9, which

suggests that items have relatively high internal

consistency.

3.2.2 Participants

197 students participated in the study and they have

ranged from 19 to 64 years of age. 48% of

participants were males and 52% were females.

Participants used two different LMS systems. 53%

of them used Moodle and 47% of them used

eCampus. Each participant attended one of eight

e courses. Each e-Course was led by one of eight

teachers.

ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE

DELIVERY

313

3.2.3 Participant Satisfaction

Participants’ opinion on e-Learning environment

(i.e. e-Learning portal powered by LMS system)

quality and e-Course quality has been measured by

descriptive statistics.

Descriptive statistics of portal quality and e-

Course quality are reported in Table 1.

Table 1: Descriptive Statistics – all Students.

N Min Max Mean

Std.

Dev.

Portal Qual. 197 1.5 5.0 3.94 .66

e-Course Qual. 197 1.4 5.0 3.93 .69

As it can be seen, participants assessed e-portal

quality and e-Course quality rather high. The

expected mean of both variables was 3 (neither

unsatisfied nor satisfied). The measured values were

almost 4 in the scale from 1 to 5.

Students who have used the LMS system Moodle

have assessed their learning experienced as shown in

Table 2.

Table 2: Descriptive Statistics - Students Using Moodle.

N Min Max Mean

Std.

Dev.

Portal Qual. 103 1.7 4.8 3.75 .57

e-Course Qual. 103 1.4 4.9 3.69 .61

As it can be seen in Table 3, those students who

have studied via LMS system eCampus have been

on average a bit more satisfied.

Table 3: Descriptive Statistics for eCampus.

N Min Max Mean

Std.

Dev.

Portal Qual. 94 1.5 5.0 4.13 .69

e-Course Qual. 94 1.5 5.0 4.19 .69

The descriptive statistics suggest a relationship

between the variables. The question is, if this

relationship is statistically significant.

3.2.4 Significant Correlation between Portal

Quality and e-Course Quality

Correlation analysis is used to verify the relationship

between numeric variables of portal quality and e-

Course quality.

The following null hypothesis is set: there is no

relationship between portal quality and e-Course

quality in blended learning setting.

Firstly, it is checked as to which test is the most

appropriate for our data?

Pearson’s correlation is appropriate if two

conditions are met: both variables are normally

distributed and that the correlation between variables

is linear.

To determine whether the variables are normally

distributed, the Kolmogorov-Smirnov test is used to

confirm normal distribution of both variables.

Scattered plot (Figure 1) shows a positive linear

relationship between both quality measures.

The results from obtaining Pearson’s correlation

are shown in Table 4.

According to the obtained results, the null

hypothesis is rejected and it can be concluded that

there is a significant relationship between the portal

quality and e-Course quality, r=.804, p (one tailed)

<0.01.

Students who are satisfied with e-Course are also

satisfied with e-Learning environment and vice

versa.

Figure 1: Scatter plot.

Table 4: Correlation.

e-Course

quality

portal

quality

e-Course Pearson Correlation 1 .804

quality Sig. (1-tailed) .000

N 197 197

portal Pearson Correlation .804 1

quality Sig. (1-tailed) .000

N 197 197

4 CONCLUSIONS

The quality of e-Learning environment and e-Course

quality is measured, in this research, with the use of,

DEMA model, which has been developed in this

CSEDU 2011 - 3rd International Conference on Computer Supported Education

314

study. Evaluation results with this model prove that

the quality of e-Learning environment and the

quality of e-Course is significantly correlated.

On an average, it can be concluded that observed

e-Learning portals show a positive attitude in

relation to their use. e-Courses have also been

assessed above expected average. From the

evaluation, it was evident that there is a pattern in

which case:

 Students who are satisfied with e-Courses are

also satisfied with e-Learning environment.

 Students who are unsatisfied with e-Courses are

also unsatisfied with e-Learning environment.

However, it is not evident from this research, which

of these two factors (e-Course, e-Learning

environment) has a greater impact on students’

satisfaction. However, the role of the teacher, who

tutored the course in e-Learning environment, is

very important. He/she is the one who facilitates

students’ learning through the use of technology.

He/she should use it in a way that technology helps

students to learn efficiently and effectively. If LMS

system is of good quality and does not cause any

troubles to students, students probably assess

technology useful, if they like the teacher’s activities

and e-Learning materials.

The experiment was carried out in case of

blended learning courses with minor face to face

part. Further research will show if there is also a

significant relationship between e-Learning

environment quality and e-Course quality in the case

of complete e-Learning settings (i.e. courses without

any face to face meetings).

In this study, DEMA model includes short

questionnaire with 28 items. In future studies we

will apply more extensive and standardised

questionnaires for evaluating students’ satisfaction

with LMS system and e-Course as well as to verify

the significance of correlation between e-Learning

portal quality and e-Learning course quality.

REFERENCES

Ardito, C., Costabile, M., Marsico, M., Lanzilotti, R.,

Levialdi, S., Roselli, T., & Rossano, V. (2006). An

approach to usability evaluation of e-learning

applications. Universal Access in the Information

Society, 4(3), 270-283. doi: 10.1007/s10209-005-

0008-6.

Bates, P., & Obexer, R. (2005). Evaluating student-

centred teaching and learning strategies for Aviation

students using a quality framework for online learning

environments. In ASCILITE 2005 conference

proceedings: Balance, Fidelity, Mobility: Maintaining

the Momentum.

Costabile, M. F., De Marsico, M., Lanzilotti, R.,

Plantamura, V. L., & Roselli, T. (2005). On the

Usability Evaluation of E-Learning Applications.

Paper presented at the System Sciences, 2005. HICSS

'05. Proceedings of the 38th Annual Hawaii

International Conference on.

Debevc, M., & Lapuh Bele, J. (2008). Usability testing of

e-learning content as used in two learning

management systems. EURODL, (1). Retrieved from:

http://www.eurodl.org/?p=archives&year=2008&halfy

ear=1&article=296

Dringus, L. P., & Cohen M. S. (2005). An adaptable

Usability Heuristic Checklist for Online Courses. In

Proceedings of the ASEE/IEEE Frontiers in Education

Conference, Indianapolis, USA.

ECDL CTP (2010). European Computer Driving License

Certified Training Professional Programme.

Eisner, E. W. (1985) The Art of Educational Evaluation.

Lewes, Falmer Press.

Kirakowski, J. & Corbett, M. (1993). SUMI: The Software

Usability Measurement Inventory. British Journal of

Educational Technology, 24, 210-212.

Kirkpatrick, D., & Kirkpatrick, P. (2006). Evaluating

Training Programs (3rd ed.). San Francisco, CA,

Berrett-Koehler Publishers, Inc.

Martin-Blas, T., & Serrano-Fernandez, A. (2009). The role

of new technologies in the learning process: Moodle as

a teaching tool in Physics. Computers & Education,

52(1), 35-44.

Nielsen, J. (1994). Usability Engineering. San Francisco,

USA Morgan Kaufmann.

Phillips, R. (2005). We can’t evaluate e-learning if we

don’t know what we mean by evaluating e-learning!

Interact, 30, 3-6. Learning Technology Support

Service, University of Bristol.

Rosenberg, M. J. (2001). E-learning: Strategies for

delivering knowledge in the digital age. New York,

McGraw-Hill.

Squires, D., & Preece, J. (1999). Predicting quality in

educational software: Evaluating for learning, usability

and the synergy between them, Interacting with

Computers, 11(5), 467-483.

Zaharias, P., & Poylymenakou, A. (2009). Developing a

Usability Evaluation Method for e-Learning

Applications: Beyond Functional Usability.

International Journal of Human-Computer

Interaction, 25(1), 75-98.

ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE

DELIVERY

315