STUDENT'S EVALUATION OF WEB-BASED LEARNING
TECHNOLOGIES IN A HUMAN-COMPUTER INTERACTION
COURSE
Dina Goren-Bar
Department of Information Systems Engineering, Faculty of Engineering Sciences
Ben-Gurion University of the Negev, P.O.Box 653, Beer-Sheva, 84105, Israel
Keywords: web-based learning, evaluation
Abstract: The hum
an-computer interface (HCI) field is constantly changing and designers are challenged to develop
simple interactive systems implemented through sophisticated technology. At Ben-Gurion University, the
introductory HCI course was originally taught in a face-to-face mode and covered theoretical knowledge on
HCI theories, principles and design, and practical experience in designing and evaluating websites. When it
became apparent from students' course evaluations that they expected the HCI course to provide them with
more hands-on experience with different types of interaction, communication devices, and design dilemmas,
the course was redesigned. The new course combines face-to-face lessons, e-learning sessions and web-
based collaborative projects. While there is still room for improvement, student's evaluations show
significant increase in satisfaction with the course.
1 BACKGROUND
The Human Computer Interaction course is taught
during the third year of a four-year program (8
semesters) in Information Systems Engineering at
Ben-Gurion University of the Negev
(http://www.ise.bgu.ac.il). The ISE curriculum is
structured to provide students with the concepts and
tools that form the fundamental base of knowledge
essential to computer information systems
professionals in today's modern technological
environment. Graduates of the ISE Department are
provided with the tools to perform diverse tasks in
the IS field such as specifying the information needs
of users and managers in organizations; performing
feasibility studies of information systems; analyzing
and designing software, data bases and user
interfaces; and developing prototype systems by
application of appropriate analysis and design
methodologies and CASE tools. Other tasks include
programming, implementation, maintenance and
administration of information systems.
Due to constant changes in the field, human-
com
puter interface designers are confronted with the
development of easier interactive systems based on
sophisticated technology. Unlike the common
courses taught in computer science and information
systems, which are topic oriented, HCI is an
interdisciplinary field. It requires the understanding
of theories and principles of design and development
of interactive systems through different
technologies. It demands also awareness to
psychological and cognitive sciences and evaluation
issues, as well as practical development skills to
implement interfaces to work properly with a variety
of users.
For several years we taught a Human Computer
Interacti
on Introductory course in a face-to-face
mode. The course provided theoretical knowledge
on HCI theories, principles and design topics and
practical experience web sites design and evaluation.
The objectives of the face-t
o-face course were:
Learn
basic concepts, theories and approaches
in the HCI field.
To de
velop a design point of view. To acquire
the necessary skills to analyze interaction
problems on a technical, a cognitive, and a
functional basis and to propose plausible
improvements based on practical guidelines,
theories and research findings.
Devel
oping a framework for orderly thinking
in formulating, clarifying, implementing and
evaluating HCI designs for interactive
systems.
206
Goren-Bar D. (2005).
STUDENT’S EVALUATION OF WEB-BASED LEARNING TECHNOLOGIES IN A HUMAN-COMPUTER INTERACTION COURSE.
In Proceedings of the Seventh International Conference on Enterprise Information Systems, pages 206-212
Copyright
c
SciTePress
Be familiar with frontier technologies, tools
and methodologies to implement and evaluate
user interfaces for different interactive
systems using the variety of tools and
methods that were presented during the
course.
The course covered the following topics:
Principles and theories of the human-
computer interactive systems
Characteristics of the human information
processing
User Centered Design and Evaluation
approaches, methodologies and tools for HCI
Software architectures and standards for user
interfaces
Interaction Styles and Design topics for web-
based systems
Designing and developing interfaces for
diverse input and output devices
Supporting user errors – Design issues,
documentation, help and tutorials
Advanced topics – CSCW, e-learning, search
and visualization
Students submitted three assignments during the
course. The first assignment focused on the
evaluation of an existent website. The second
assignment required the design of a new website
applying the principles of design acquired during the
course. In this assignment students supplied the
rational for designing the new website and
performed a "hands on" exercise using standard
development tools (visual basic, Java script,
HTML). The third assignment required the
assessment of the new website (performed in the
second assignment) implementing methodologies of
usability testing. Students selected a partner to
perform the assignments.
Students attended classes once a week. During
the classes the tutor (a senior staff member) taught
the course topics assisted by PowerPoint
presentations. Students' interaction with the lecturer
was mainly through questions asked during the
class. A course assistant (a graduate student) was
available during reception hours. The assistant
helped students according to students' initiative.
The course had an additional website. The
website included: learning materials (presentations),
the syllabus and assignments instructions.
Annual students' evaluations revealed that they
expected from the HCI course to provide them with
more hands-on experience on different kinds of
interaction, communication devices and design
dilemmas than those acquired during the course.
Besides, they expected the course to be more
"interesting". During interviews conducted with
volunteer students they expressed their willingness
to experience new tools and modes of interaction in
addition to those taught during the course, such as
collaborative applications and e-learning.
We decided to redesign the Human Computer
Interaction introductory course intending to meet
these expectations.
2 THE NEW COURSE
The new Human Computer Interaction introductory
(HCI 2002/3) course was taught at the Information
Systems Engineering Department in Ben-Gurion
University. The course objectives and topics were
the same as those in the former course (see prior
section). The HCI 2002/3 course had a new major
improvement; the addition of a website which
included a collaborative workspace that allowed the
students to interact with other fellow students and
the course staff. The course site enabled the students
to experience different kinds of interactions with
various technologies such as e-learning and
collaborative environment; to practice the theoretical
principles and the technology involved; to design
prototype systems implementing both their personal
experience and the theoretical and technical
knowledge acquired during the course. The course
combined three kinds of learning methodic: (1) face
to face lessons; (2) web-based lessons and (3) web-
based design and collaborative evaluation projects.
A hundred and fifty students have participated in
this course.
We will describe the implementation of web-
based lessons and collaborative technologies in the
HCI 2002/3 course and present the students'
evaluation results.
2.1 Implementation of interactive
lessons in the HCI 2002 course
The HCI 2002/3 course was delivered in the Mixed
Mode (Harasim, 2000a, 2000b) (where students
participated in F2F (face to face) lectures and
Interactive Distance Learning techniques. The
Mixed Mode of web-based learning employs
networking as significant portion of a traditional
classroom or distance course. It is distinguished
from the adjunct mode (which typically refers to the
use of the network as an enhancement but not as a
required or graded component of course activity) by
the fact that networking is fully integrated into the
curriculum. The networking activities constitute a
regular part of the course and are included on the
course grade. Mixed mode learning has many
variations. It may be used for one or more major
STUDENT'S EVALUATION OF WEB-BASED LEARNING TECHNOLOGIES IN A HUMAN-COMPUTER
INTERACTION COURSE
207
activities in a traditional face-to-face or distance
mode course, such as small group discussions,
seminars, and group projects. An example is an
undergraduate course in Communication at Simon
Fraser University, Canada
(
http://www.sfu.ca/index3.htm) that uses six weeks
of a thirteen-week course for online student-led
seminars; the other seven weeks are held as face-to-
face lectures and tutorials. The use of online
seminars in a face-to-face class is a common
application of mixed mode delivery. Usually, online
seminars enable all students to participate,
something impossible in a small face-to-face
classroom, and certainly impossible in large
undergraduate classes.
Harasim (2000) reported from data collected in
the Virtual-U project (http://virtual-u.cs.sfu.ca/
vuweb.new/vuproduct.html) that in mixed mode
courses, the level of active student participation and
interaction is significant.
As in the Communications course held at Fraser
University (Ramiller, 2002), in the new HCI 2002/3
course six weeks of a thirteen-week course were
conducted as web-based lessons + virtual
collaborative projects; the other seven weeks were
held as face-to-face lectures and tutorials.
Students interacted on a one-to-one basis on the
interactive lessons in the course site. The course had
six interactive lessons. Six topics, one each week,
were learned in each interactive lesson. Students
accessed the interactive lesson through the course
website. In each interactive lesson a new topic was
presented and students were requested to perform
various tasks and to answer questions based on the
studied topic. For each topic, the students were
required to read relevant articles, book chapters or
web material linked through the interactive lesson.
The objective was to provide the students with basic
knowledge and understanding on the concepts and
ideas discussed during the interactive lessons.
The overall length of an interactive lesson was
approx. 30 minutes. However, to complete all the
tasks, usually students started an e-lesson and
stopped after a while. It was not allowed to
introduce changes to completed tasks (to diminish
plagiarism). Each time a student came back to the
same lesson s/he was able to continue from the place
they stopped.
The topics learned through the interactive
lessons were assessed with diverse techniques.
Students defined concepts, searched for relevant
examples and uploaded their URL, proposed their
own solutions and answered open and multiple-
choice questions. All the user data and behavior
during the interactive lesson was recorded. The
course assistant checked the open questions, while
closed questions were checked automatically,
rendering a combined grade for each lesson. Finally
each student got a mark for all the interactive
lessons.
Students were required to fill a web-based
feedback questionnaire for each lesson. This task
was voluntary. We encouraged students to fill it;
however, no grades were given for this task in order
not to bias results.
After finishing the lesson the student got an
email confirming that all the tasks on that lesson
were submitted. After grading the whole lesson, an
automatic mail was sent to the student with the
corresponding grade.
2.2 Implementation of Collaborative
Assignments in the HCI 2002/3
course
The standard one to one interaction between user
and computer has been challenged during the last
years by the development of distributed and
collaborative applications. Collaborative learning is
defined as a learning process that emphasizes
cooperative efforts among faculty and students. It
stresses active participation and interaction by both,
students and instructors (Bouton. & Garth, 1983;
Bruffee, 1984).
The collaborative learning approach is
considered an interactive approach (Alavi, 1994). It
can be treated as a method that encourages students
at various performance levels to work together
toward a common goal (Johnson, 1981; Johnson &
Johnson, 1975). Harasim (1990) indicates that
“Collaborative learning” is fundamentally different
from the traditional “direct-transfer” or “one-way
knowledge transmission” model in which the
instructor is the only source of knowledge or skills.
In collaborative learning, instruction is learner-
centered rather than teacher-centered and knowledge
is viewed as a social effort, facilitated by peer
interaction, evaluation and cooperation. Therefore,
the role of the teacher changes from the transferring
of knowledge to students to being a facilitator in the
construction of the student's own knowledge (Hiltz
& Benbunan-Fich, 1997).
Asynchronous Learning Network (ALN’s) is a
teaching and learning environment located within a
Computer-Mediated Communication (CMC) system
designed for anytime/anyplace use through computer
networks (Hiltz, 1994; Hiltz & Wellman, 1997). The
asynchronous nature of the interaction leads to new
paradigms for teaching and learning. They state that
the most important element for an ALN learning
environment that supports collaborative learning and
relates to the “social construction” of an interaction
ICEIS 2005 - SPECIAL SESSION ON EFFICACY OF E-LEARNING SYSTEMS
208
environment is the appropriate expectation and
norms of interactions. In particular, the instructor
role must be re-conceptualized. The instructor needs
to encourage students to look upon their interaction
with their peers as valuable resources for learning,
rather than focusing on memorizing lecture-type
material presented by an instructor.
Ramiller (2002) points out that the use of a
project assignment helps to promote active learning
through hands-on engagement within a collaborative
learning context. He implemented an innovative
approach, the Virtual Interactive Project (VIP) that
explores the middle ground between field projects
and text-based projects in an effort to achieve some
of the advantages of both. The project evolved
through web-based and email interaction between
students and a "virtual client" representing the firm
in the case.
Beyond the common principles, which guide the
development of all human computer interfaces,
specific principles involve the development of the
same interfaces to enable collaboration through
different devices. However, most of the students
have no prior experience with those technologies as
users. We intended to provide the students with the
experience of cyber-work in the web with different
tools.
We implemented an Asynchronous Learning
Network (ALN’s) as a teaching and learning
environment, which included e-mail, public
conference (forum) and private conference facilities
(suitable for multiple group projects) (Clark, 2000,
Harrasim, 1991; Lehtinen et al., 2000) to enable
active collaboration of several group members
participating in the same assignment. All the
interactions within the forums were saved in the
course database.
The course population (150 students) was
divided into 15 groups of 5 pairs of students each
one (ten students in each group). The students
performed two collaborative assignments, with the
same structure but on different topics. Students had
five weeks to elaborate each one of the two
collaborative assignments and perform all tasks. In
the first assignment, each group received a
description of a different desired system or tool. The
objective of the first assignment was experiencing
with the phases of a user centered design project.
Each phase was assigned to one pair of students. The
first pair was responsible for performing the
exploration phase; the second pair gathered and
analyzed the user requirements. The third pair was in
charge of the design and functionality of the system.
The forth pair defined the usability testing based on
Nielsen’s usability parameters (Nielsen, 1993). The
last pair prepared a comprehensive presentation that
summarized all the pairs’ activities and products and
presented it to the class in a face-to-face meeting.
The objective of the second assignment was to
create an interactive lesson on the web. The
assignment was divided into 5 tasks were each team
was in charge of a different task, same as in the first
collaborative assignment. The assignment’s tasks
were either technical such as: building a DB,
developing a website or code writing for the
interactive lesson or conceptual tasks e.g.
exploration, bibliography review, writing and
designing the lessons’ content. Each group prepared
an interactive lesson on a different subject.
In addition, it was clearly emphasized by the
course instructors that the group members must
collaborate, as the product of one task will act as a
building block for the following tasks.
To facilitate communication between students,
the course website interface was developed in
Hebrew. In order to enable collaboration between
the group members, each group was provided with a
workspace within the course site. At the group
workspace a full description of the project was
available. All students in the same group had to be
acquainted with all the subtasks of the assignment.
For each one of the two group assignments, each
pair within the group had to select a task from the
list of available tasks, via the collaborative
environment. Students had two roles: as sub-task
leader for one week, and as other students' tasks
reviewer for the rest of the time. While performing
as subtask leaders, students work in small teams,
with one peer, to prepare and present the topic,
moderate the discussion for a week and upload a
written assignment on their sub-task. When a
document was uploaded all the members of the
group received an automatic message from the
system including the name of the team, which
submitted a file, and the name of the file. Students
uploaded articles, pictures, sound files, forms and
excel sheets among others. Other students had to
read and review the files and comment on them.
There was no limitation on the number of working
files each subtask team could upload. However, at
the submission deadline, just one integrated file was
permitted for each subtask.
STUDENT'S EVALUATION OF WEB-BASED LEARNING TECHNOLOGIES IN A HUMAN-COMPUTER
INTERACTION COURSE
209
3 STUDENTS EVALUATION OF
THE HCI 2002/3 COURSE
TEACHING METHODOLOGIES
3.2 Methodology
After each lesson students were requested to fill an
online feedback form at the course website. The
questionnaire scale range is: 1 (totally disagree) - 5
(totally agree). Form filling was encouraged but
voluntary. In addition, a face-to face open feedback
discussion was conducted at the end of the two
collaborative assignments students' presentations.
3.1 Goals and Rationale
The purpose of the evaluation was to test students'
satisfaction from the different teaching
methodologies implemented in the course. We
decided to test mainly students' preferences, a
subjective measurement, because we had no
influence on their opinion. Moreover, being this
course one with many tasks to perform each week,
requiring considerable student's responsibility, this
fact may have a negative influence on student's
course evaluation. We opted for this unfavorable
situation instead of relying on grades evaluation.
Basing our evaluation on course grades may have
been unintentionally biased, considering that those
who assigned the grades are the course teaching
staff.
The following is a summary of the feedback
questionnaire. The total number of questionnaires
filled was 678.
The average participation per lesson type is the
mean rate of voluntary evaluation form- filling
relative to the total number of students enrolled in
the course (150) not related to other types of lessons.
Table 1 show that students rated the e-learning
sessions three times more than the F2F lessons and
the collaborative learning.
We expected the students to have different
preferences towards the three types of lessons (see
Table 2). We grouped the negative evaluations (1-2)
and the positive ones (4+5). Overall answers to all
the questionnaires were 8126 (approx. 677*12). We
performed Chi2 test of association. We found that
students preferred the F2F lessons significantly more
than expected, compared to the e-learning lessons
(Chi2 26.88, df= 4, p<0.001). However when we
analyze all the rates given to the e-lessons alone,
most of the students significantly preferred to study
HCI topics through web based learning (Chi2 for
goodness of fit test= 247.7848, df=2, p< .001) .
Nevertheless we compared the final grades
distribution from this course with former ones (two
years ago) and found no significant difference.
Students read the same bibliography (in the mixed
model course they read added references). The grade
components were different; therefore no other
comparison was possible.
This comparison was preformed as a validity
test. Having similar marks, the question remains:
What did the students preferred? How did they
evaluate the different learning experiences they got?
And, did they enjoy the learning process more?
Table 1: Average Percentage of Evaluation Form filling by Lesson Type
Lesson Type
Average
participation per
lesson type
Relative percentage of
answers per
all feedbacks (678)
F2F Lessons 21.50% 19.02%
Interactive Lessons 66.00% 73.04%
Collaborative Lessons 18.00% 7.94%
ICEIS 2005 - SPECIAL SESSION ON EFFICACY OF E-LEARNING SYSTEMS
210
Table 2: Students' evaluation toward the three types of lessons.
Evaluation F2F
lessons
Interactive
Lessons
Collaboration
lessons
Total
answers
Disagree (1-2) 315 1543 160 2018
Neutral (3) 480 1879 202 2561
Agree (4-5) 753 2518 276 3547
Total Answers 1548 5940 638 8126
Chi2 = 26.88450469, df= 4, p<0.00
4 DISCUSSION AND
CONCLUSIONS
The revised HCI introductory course for Information
Systems Engineering was first introduced during
winter 2002/3. It combined face-to-face lessons, e-
learning sessions and web-based design
collaborative projects. The philosophy behind the
course was to give the students, besides the domain
knowledge, the opportunity to experience new
technologies by performing practical work through
systems they might design in the future (such as e-
learning and collaborative work).
Student's evaluations showed significant
satisfaction from the new course methods, however,
new methodological and practical problems should
be considered.
First, it is important to mention that this kind of
course requires a considerable effort from the course
academic, administrative and technical staff. The
course involves three phases of development:
preparation, testing and delivery.
All the course material should be prepared in
advance. Besides the presentations for the F2F
lessons, the web lessons require many hours of
preparation. The collaborative environment is a full
development project demanding continuous
technical support before and during the course
delivery.
Web lessons should be tested as well as the
collaborative environment. Communication
difficulties may appear unexpectedly, disturbing the
course schedule.
During the course delivery, the course staff is
engaged many hours with course activities. Students
are active beyond working hours, during weekends
and holidays, expecting from the course staff to
respond in a timely fashion.
Students ranked highly the F2F lessons.
Although this result is based on a relatively small
number of students that voluntarily filled the
students feedback forms (an average number of 32
students graded the F2F lessons), it is corroborated
with the high participation rate in these lessons (no
mandatory assistance was required). The course
presentations were public at the course website;
therefore, it seems that students appreciated the F2F
discussions involving personal and social
interaction. This result should be verified in future
studies with a larger number of students.
One of the most exciting results in this study was
the high degree of voluntary feedback filled up by
the students to the e-learning lessons. Students were
encouraged to do so, however, no grade reward was
accredited. An average of 99 students (from 150)
filled up their feedback to each e-learning lesson.
Moreover, Chi2 for goodness of fit test showed that
more students evaluated these lessons significantly
higher than expected (p< .001).
Student expressed that the course material was
well organized and it was easy to track course
presentations on the web.
A smaller number of students evaluated the
collaborative lessons. There are several possible
reasons for this finding. The first, and most
probable, is that students had an opportunity to give
F2F feedback by the end of each collaborative
assignment presentation lesson. Student expressed
that it was easy to communicate and perform the
required tasks within the virtual environment. The
feedback on the collaborative assignments was that
at the beginning it was hard to accept that they have
to cooperate in order to get the work done. Few
students felt it as "unfair" since dedicated
responsible students should rely on inputs of less
successful ones. During the process the students
realized that the only way to feel more confident
with other's inputs is to be in touch, to give feedback
STUDENT'S EVALUATION OF WEB-BASED LEARNING TECHNOLOGIES IN A HUMAN-COMPUTER
INTERACTION COURSE
211
on each other's documents or outputs and check if
the feedback was reflected in the revised documents.
Few students questioned the need for this kind of
assignment due to its complexity. Other students
replied that in order to succeed in a project in their
workplaces (mostly technological and software
companies) they have to cooperate within their
teams and sometimes with other teams. They stated
that collaborative assignment encouraged individual
thinking as each team is in charge on leading a task
and in addition, each team is committed to the
success of the whole assignment. Another feedback
was on the group structure, students complained that
they were missing the instructor role while other
groups replied that they have nominated a group
chair that was in charge on the timeline and to
encourage the group members. Most of the students
expressed that during the first assignment they
learned how to collaborate and that the second
assignment was easier on this aspect (but more
difficult on the subject matter).
Another possible reason for the low rate of
feedback form-filling on collaborative lessons is that
students gave feedback to each other within the
collaborative environment many times during the
assignment period. They might have felt that they
did their job already.
It is worthy to notice that few students didn't like
the e-learning or the collaborative lessons. This
finding may have two possible implications about
the HCI 2002/3 course. One the one side, the course
does not match all types of students. Some students
may prefer to work alone. For those students the
collaborative assignment may be very difficult. On
the other side, the HCI 2002/3 course is very varied,
enabling students with different learning styles to
perform better in their preferred kind of lesson.
We conclude that the HCI 2002/3 course enabled
students to achieve several goals: to learn theoretical
principles, technologies, development and
evaluation methodologies through the F2F and web-
based lessons as well as practicing the topics learnt
through discussions and implementation within the
collaborative environment. It seems that this kind of
course has many advantages as well as several
drawbacks that should be considered before
adopting the mixed model.
REFERENCES
Alavi, M. 1994. Computer-Mediated Collaborative
Learning: An Empirical Evaluation, MIS Quarterly,
Vol. 18, No. 2 159-174.
Bouton, C. & Garth, R.Y. (Eds.) 1983. Learning in
Groups, New Directions in Teaching and Learning,
No. 14, Jossey-Bass Inc., San Francisco, CA.
Bruffee, K. A. 1984. Background and history to
collaborative learning in American college, College
English, Vol. 46, No. 7 635-652.
Clark, J. 2000. Collaboration Tools in Online Learning
Environments http://www.aln.org/alnweb/
magazine/Vol4_issue1/Clark.htm.
Harasim, L. Ed. 1990. On-line Education: Perspective on
a new medium, Praeger/Greenwood, New York, NY.
Harasim, L. 1991. Designs and tools to augment
collaboration in computerized conferencing systems,
Proceedings of the Hawaiian International
Conference on Systems Science, Vol. 5 379-385.
Harasim, L. 2000a. The Virtual University: A State of the
Art in Advances in Computers, Book Series, Vol. 54,
Academic Press.
Harasim, L. 2000b.
Shift Happens: Online Education as a
New Paradigm in Learning
, Internet and Higher
Education: Special Issue, Vol. 3, No. 1. Elsevier
Science, UK 41-61.
Hiltz, R. 1994. The virtual classroom: Learning without
limits via computer networks, Human-Computer
Interaction Series.: Ablex Publishing Corp. New
Jersey .
Hiltz, S.R. and Benbunan-Fich, R. 1997. Supporting
Collaborative in Asynchronous Learning Network,
invited keynote address for the UNESCO/ Open
University Symposium on Virtual Learning
Environment and the Role of the Teacher, NJ Inst.of
Tech. http://eies.njit.edu/~hiltz/CRProject/unesco.htm.
Hiltz, R. and Wellman, B. 1997. Asynchronous learning
networks as a virtual classroom, Communications of
the ACM, Vol. 40, No.9 44-49.
Johnson, D.W. 1981. Student-student interaction: the
neglected variable in education, Educational
Research, Vol. 10, N.1 5-10.
Johnson, D.W. and Johnson, R.T. 1975. Learning together
and alone: Cooperation, competition and
individualizations, Englewood Cliffs, Prentice Hall,
New Jersey.
Lehtinen, E., Hakkarainen, K., Lipponen, L., Rahikainen,
M., & Muukkonen, H. 2000. Computer Supported
Collaborative Learning: A review, In H. Meijden, R.
Simons, & F. de Jong (Eds.), Computer supported
collaborative learning in primary and secondary
education. Report for the EC, Project 2017, University
of Nijmegen.
www.kas.utu.fi/papers/clnet/
clnetreport.html
1-46.
Nielsen, J. 1993. Usability Engineering, AP Professional.
23-37.
Ramiller, N.C. 2002. The Virtual Interactive Project:
Teaching Analysis and Design Through Narrative and
Drama, Communications of the Association of
Information Systems, Vol. 9, No. 1, July.
ICEIS 2005 - SPECIAL SESSION ON EFFICACY OF E-LEARNING SYSTEMS
212