Evaluation of E-Learning in Indonesian as a Foreign Language
Setya Tri Nugraha
1,2
, Sarwiji Suwandi
1
, Joko Nurkamto
1
and Kundharu Saddhono
1
1
Universitas Sebelas Maret, Surakarta, Indonesia
2
PBSI, FKIP, Universitas Sanata Dharma
Keywords: E-Learning, Program Evaluation, Improvement, CIPP.
Abstract: Planning, implementing, and evaluating an e-learning program can be a complex task because e-learning
involves multiple dimensions and aims to meet the needs of students and e-learning program stakeholders.
This study aimed to describe and evaluate the context, input, process, and product of e-learning program.
The CIPP Model was used as the evaluation approach to systematically guide the conception, design,
implementation, and evaluation of the e-learning program, and provide recommendations of the program’s
effectiveness for continuous improvement. The data were gathered through interview, observation, and
examination of written documents. Content analysis was carried out to analyse the qualitative data. Results
of the study indicated that the e-learning program to teach Indonesian as a foreign language needs to be
improved in some dimensions. The findings revealed that some improvements in the physical conditions, e-
learning artefact, e-learning objects, technology, interface design, and evaluation dimensions of the e-
learning program were required to make the program more effective in the future.
1 INTRODUCTION
The success of the e-learning program is determined
by various interrelated factors. Fitzpatrick (2012)
states that there are five key factors of success in e-
learning, namely: (1) human issues: the readiness of
the instructor and teacher to design e-learning; (2)
technical skills: instructor’s competence in IT to
interact with the learners; (3) technical support: the
existence and function of technology to ease access
to communicate among the instructors and learners;
(4) collaboration: the highest level of readiness
among the instructors, learners, and curriculum
developer to build communication and collaboration;
(5) attitude: strong belief in e-learning manifested in
the mindset and positive attitude towards the entire
process in the e-learning program.
Other than the five factors mentioned earlier,
there are eight dimensions which determine the
success of the e-learning program. The eight
dimensions include: (1) institutional: this dimension
is related to the administrative readiness of the
institution in conducting an e-learning program; (2)
pedagogical: this dimension covers the learning and
teaching process which is supported by curriculum
specification which is suitable with the
characteristics of e-learning; (3) technology: this
dimension is related to the readiness in terms of
technology and its infrastructure to ensure the
various forms of communication in e-learning; (4)
interface design: this dimension is focused on the
synchronous and asynchronous interaction in the
learning process; (5) evaluation: this dimension
deals with the evaluation process of the learning
process and program evaluation; (6) management:
this dimension leads to the efficiency and
effectiveness in the management and distribution of
information; (7) source support, this dimension deals
with the availability of various technological sources
to ease access of communication among the
components of e-learning; and (8) ethics: this
dimension covers the learner’s socio-cultural context
and ethics in using technology (Khan, 2005a; Khan,
2005b; Khan, 2005c).
To find out the level of success of those factors,
a systematic program evaluation is called for.
Program evaluation is understood as a process to
determine the degree of utility of the object being
evaluated and to give recommendation to enhance
the program (Patton, 2002; Lodico, Spaulding, and
Voegtle, 2010). There are several evaluation models
which have been used to find out the achievement of
an educational program, such as: the Tyler Model,
Scriven, PTE Model, decision-making evaluation
Nugraha, S., Suwandi, S., Nurkamto, J. and Saddhono, K.
Evaluation of E-Learning in Indonesian as a Foreign Language.
DOI: 10.5220/0007169604810486
In Proceedings of the Tenth Conference on Applied Linguistics and the Second English Language Teaching and Technology Conference in collaboration with the First International Conference
on Language, Literature, Culture, and Education (CONAPLIN and ICOLLITE 2017) - Literacy, Culture, and Technology in Language Pedagogy and Use, pages 481-486
ISBN: 978-989-758-332-2
Copyright © 2018 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
481

approaches, naturalistic evaluation approaches,
model Kirkpatrick (Owston, 2008; Fitzpatrick,
Sanders, and Worthen, 2012). In this research, one
of the decision-making evaluation approaches, i.e.
CIPP Model developed by Stufflebeam (2003;
2014), was selected.
The e-learning program chosen as the object of
this research is the Indonesian Language e-learning
program for Foreign Speakers (in Indonesian, it is
called BIPA e-learning) developed by Wisma
Bahasa, one of the language training institutions for
BIPA in Yogyakarta. The e-learning program in WB
is understood as the planned learning experiences
using a wide spectrum of technology, especially
internet to interact with learners who are separated
by time and distance.
This study aimed to describe and evaluate the
context, input, process, and product of e-learning
program in WB Yogyakarta and provide
recommendations of the program’s effectiveness for
continuous improvement. The framework of this
CIPP evaluation model is used systematically to
evaluate the planning, implementing and evaluating
of e-learning program in WB Yogyakarta in order to
improve the program and to provide inputs for the
sustainability of the program.
2 CIPP EVALUATION MODEL
The model used in this evaluation is the Context,
Input, Process, Product Model (CIPP) developed by
Stufflebeam (2003). In this model, evaluation is
interpreted as a systematic investigation based on
certain criteria of the planning stage to the delivery
of an innovative program to assess the values,
usefulness, and functionality of an object or product
of knowledge so that the information obtained in the
process helps the decision making (Stufflebeam,
Coryn, and Chris, 2014; Joint Committee on
Standards for Educational Evaluation, 1994).
The CIPP evaluation model was used by various
experts in education. Tiantong and Tongchin (2013)
used the website-based collaborative learning model
employing Context, Input, Process, Product (CIPP)
evaluation model developed by Stufflebeam. The
research focused on the learning with multiple
intelligences supported by internet-connected
devices for collaborative learning. The evaluation
coverage was directed to context, input, process and
product. The context evaluation covers socio-
political investigation, organizational and contextual
variables related to the instructional needs using the
Internet. The input evaluation was directed to
comparing various input alternatives to fulfill the
learners’ needs, including website-based instruction.
The process evaluation was carried out formatively
in the instructional planning, design, development
and implementation aspects to improve the process.
Meanwhile, the product evaluation was focused on
the summative evaluation to assess quality,
usefulness, and instructional values of the program.
Studies using Stufflebeam’s CIPP model to
evaluate curriculum was also conducted by Hakan
and Seval (2011). The purpose of this study is to
determine the validity and reliability of the
evaluation scale developed based on the principles
of Stufflebeam’s CIPP model in the context of
English curriculum evaluation in Yildiz Technical
University.
The CIPP evaluation model was also used by
Oluwatobi (2015) to evaluate context, input, process,
and product in the Postgraduate Program of Babcock
University. The results of the study showed that the
postgraduate program in Babcock University
experienced rapid improvement in terms of learning
patterns among the students as indicated by their
understanding of the program visions in the research
and learning processes.
The important concepts of CIPP Model were
context, input, process, and product. The four
concepts describe four types of evaluation activities.
The context evaluation refers to the activities of
problem assessment, needs and its possible
emergence in educational program settings. The
decision makers used context evaluation to
determine the goals and priorities to ensure the
attainment of the program objectives. Stakeholders
may later use the results of the context evaluation to
assess whether the program being implemented is in
accordance with the needs and problems that have
been determined since the beginning of the program.
The input evaluation refers to the identification
and assessment of various approach alternatives
used in implementing the program, assessing the
procedural planning, budgeting, and staffing
provision to match the target needs and outcomes of
the objectives. Essentially, input evaluation must
involve the identification process of various
approaches using various types of instruments and
materials to be used in the learning process and to
help the decision-makers to prepare the selected
approach to be implemented in the learning stage.
Process evaluation refers to monitoring,
document evaluation, and activity assessment during
the implementation of the program. The activity of
the evaluation process includes examining the
implementation of the plan and its documentation. In
CONAPLIN and ICOLLITE 2017 - Tenth Conference on Applied Linguistics and the Second English Language Teaching and Technology
Conference in collaboration with the First International Conference on Language, Literature, Culture, and Education
482

the process evaluation, the evaluator makes
comparison and contrast among the existing
activities in the implementation and planning stages,
describes the emerging problems during the
implementation, and assesses the suitability of the
program ranging from planning documents and
documenting the implementation.
Meanwhile, the product evaluation was
emphasized on testing the impact of the program on
the target audience, the quality and importance of
outcomes, and how far the program is disseminated.
The CIPP evaluation model is a comprehensive
evaluation model because the aspects being
evaluated include the goal setting, planning,
action/process and outcomes (Stufflebeam, 2003).
3 METHODS
This research is an evaluation research focusing on
the e-learning program evaluation. The program
evaluation research is an evaluation design or
procedure to gather and analyze data systematically
in order to find out the degree of significance or the
practical value of the program based on certain
standards or criteria.
Research data were obtained from (1) interview,
(2) document study, and (3) observation. The
interview was conducted with focus group
interviews (Lodico, Spaulding, and Voegtle, 2010).
Document studies were conducted to determine the
quality of lesson plan documents, teaching materials,
and assessment instruments uploaded on the BIPA e-
learning program website and used by BIPA
program developers. Observations were made to
collect data accurately and naturally which reflect
the reality of the situation as observed by researchers
(Lodico, Spaulding, Voegtle, 2010). The
Observation Sheet was used to document the
learning BIPA e-learning process related to the
availability and implementation of planning,
learning resources, media, and classroom
interactions. In the meantime, field notes contained a
variety of information that researchers found during
research related to the research problems.
4 RESULTS AND DISCUSSION
4.1 Context Evaluation
Context evaluation in this BIPA e-learning program
is focused on determining the relevant contexts of
BIPA learning globally, identifying BIPA learners
with issues related to needs, and settings of learning.
In details, there are four main things in the context
evaluation, namely: 1) the availability of information
on technology-related needs and assets used in e-
learning programs, 2) the availability of information
from program stakeholders related to the potential
problems which need immediate solution, 3) the
clarity of program goals and the supporting data
which have been adjusted to the needs and potentials
of the program benefits, and 4) the availability of
such programs of the same type to be used as a
reference for the model development being
evaluated/benchmarked.
The development of BIPA e-learning program in
WB Yogyakarta is done by analyzing the needs of
BIPA learners to know the purpose, initial
proficiency, choice of topic, choice of strategy, and
their technological skills. This needs analysis is done
by interviewing and giving a pre-test or placement
test. Based on the needs analysis, the instructor
develops the material, determines the methods and
techniques, the media that suit the needs and
conditions of the learners, including developing its
systems and assessment instruments.
The results of the needs analysis related to
technological assets show that Wisma Bahasa is still
in the process of seeking and finding an adequate
form of technology to conduct e-learning programs.
The multimedia-based computer devices have
already installed software such as Skype as the main
software to implement e-learning. The software is
chosen to ensure that the learning interactivity runs
smoothly. The full feature supports in the Skype
software facilitate teacher interaction with learners
and enable the distribution of learning materials and
media. Teachers at Wisma Bahasa optimize desk
sharing facilities so that the various materials and
media used by teachers can be accessed directly by
learners without any difference in formats.
However, there are still some potential problems
in the implementation of the e-learning program,
namely: 1) the unavailability of technology format
that can bring the real-life classroom atmosphere in
the e-learning process, 2) the unavailability of
adequate class to implement an ideal e-learning
classroom (the available classes are too small and
not sound-proof), 3) the specific curriculum for e-
learning programs has not been systematically
arranged, 4) the learners’ lack of technology literacy,
and 5) internet connection interruption.
The implementation of needs analysis and the
technology asset and the identification of potential
problems which might arise are necessary to be done
to clarify the purpose of BIPA e-learning program,
Evaluation of E-Learning in Indonesian as a Foreign Language
483

i.e. to facilitate the BIPA learners who want to study
the Indonesian language but cannot come to
Indonesia to study through various planned learning
experiences using a wide spectrum of technology
utilization especially internet to interact and to
improve their Indonesian language skills.
The implementation of e-learning in Wisma
Bahasa is not specifically based on benchmarking a
similar program in certain institutions. Wisma
Bahasa tries to develop this program by trial and
error to discover the format that suits the
institution’s capacity.
The needs analysis step is in accordance with the
study of Azimi and Rahmani (2013) which affirms
that the needs assessment is an important
investigative stage that an institution or language
training provider must perform to identify the needs
and actual gaps of the learners. This strategic step
needs to be done to obtain a variety of information
about the objectives, content, linguistic background,
choice of learning styles, teaching strategies,
motivation of learners, outcomes, and also the
commitment of learners in following a language
program.
WB has a strategic vision for the future in the
education and training of Indonesian as a foreign
language, namely the internationalization of the
Indonesian language through e-learning technology.
This provides a strong foundation for the planning
and implementation of BIPA e-learning in WB, as
well as a reference for the provision of curriculum
and learning media associated with it in the context
of creating e-learning environments.
4.2 Input Evaluation
The focus of input evaluation on BIPA e-learning
program in WB capability system and alternative
selected program strategy. There are three main
objects of the evaluation, namely: 1) the availability
of curriculum, syllabus, and materials suit the
characteristics of the e-learning program, 2) the
clarity of the development strategies of the e-
learning program, and 3) the clarity of scheme in
new technology utilization in the e-learning
program.
Pertaining to the availability of a curriculum,
syllabus, and materials, WB is still looking for an
appropriate model and its components for the e-
learning program. The specific and comprehensive
curriculum document is not yet available. The
curriculum and syllabus for the e-learning program
still use the curriculum for regular programs
conducted on-site at WB. Most of the materials,
contents, and media are the same as those provided
for onsite learners who take classes at WB. The
available syllabi are those developed by the
instructors as a follow-up step after conducting
learners’ needs analysis. These syllabi are used as
the basis to facilitate the learners to achieve goals
during the e-learning programs. In relevance to the
input in the form of curriculum for BIPA e-learning
program in WB, there is no document of standard
curriculum for e-learning programs.
This strategy developed to achieve the goals of
BIPA e-learning program at Wisma Bahasa is to
bring the real-life classroom atmosphere using
various supporting variables in the BIPA e-learning
process. With this strategy, it is expected that
learners will feel that they are actually in the
classroom to interact with the teachers and/or other
learners despite the fact that they interact through
the cyberspace.
The strategy adopted is supported by the scheme
of utilizing new technology in the e-learning
program at WB by installing an e-learning
technology software such as Skype and its features.
The further projection is to develop the learning
management system specifically as it is
implemented in the onsite learning, called Sistem
Informasi Manajemen Siswa (SIMS). It is expected
that the scheme of new technology utilization will
further improve the quality of the e-learning
programs.
E-learning system with various dimensions and
media delivery has been developed in WB since
2009 until now. The principle developed in the
BIPA e-learning program in Wisma Language is to
present the classroom atmosphere with various
supporting variables in the learning process of BIPA
in e-learning. Based on this principle, learners are
expected to be present in the classroom and interact
with teachers and / or other learners even though
they interact in the cyberspace.
At this stage also, the interactivity of technology
to be used is also identified, especially the capacity
of the Internet both in WB and the learners’ location.
This is done to ensure that the interaction between
learners and teachers and with learning resources
can take place effectively and efficiently. From this
process, some obstacles were encountered related to
the learners’ varied technological literacy; some
learners were less technologically literate, so it took
more time to introduce the technology tools and
systems implemented in the e-learning program.
Another problem that arose was the occasional
interruptions of internet connection when the
learning process was in progress. This problem was
CONAPLIN and ICOLLITE 2017 - Tenth Conference on Applied Linguistics and the Second English Language Teaching and Technology
Conference in collaboration with the First International Conference on Language, Literature, Culture, and Education
484

anticipated through intensifying connection checks
before and during the learning process.
To ensure a smooth learning interactivity, WB
developed the Skype program by utilizing the share
desktop facility in this e-learning program so that the
various materials and media used by teachers can be
accessed directly by the learners without any
difference in formats. This principle is also the basis
for the use of a communicative approach in the
BIPA teaching and learning process in WB by
emphasizing on learner-centered learning with
various forms of assistance to help learners use
various language skills for several different
purposes.
The use of various tools in this e-learning
program is in line with Ehler and Hilera (2012) who
confirms that e-learning programs must utilize
various technologies to support the learning process.
It is this technology that connects learners with other
learners and diverse learning resources.
4.3 Process Evaluation
The basic orientation of the process evaluation of the
BIPA e-learning is to identify or predict the impact
of procedural design or program implementation,
provide information for decision-making related to
the learning process, provide affirmations on
ongoing learning activities, and assess the continuity
of teaching and learning activities. In detail, the
process evaluation includes: 1) the clarity of the
program management design, 2) the availability and
functioning of the e-learning artefacts, 3) the
suitability of the e-learning process being
implemented with the goals and strategies, 4) the
functioning of the supporting technology to facilitate
multi-directional interaction, 5) the contribution of
e-learning environments in achieving learning
objectives, and 6) the creation of the interaction
patterns among e-learning environments according
to the initial design.
The e-learning program management design in
WB is manifested in the package system which must
be taken by BIPA learners. The 10-package sessions
must be completed within a month so that the
institution can monitor the learning progress of the
learners to achieve competency. This requirement is
applied to meet the minimum standards of program
quality.
The program management design is supported by
the existence of e-learning supporting artefacts and
technology which facilitate the BIPA learners at WB
to interact with each other. The BIPA e-learning
process in WB is done by real-time and
simultaneous interaction as well as delayed
interaction. Real-time and simultaneous interactions
are made with Skype programs supported by share
desktop programs to facilitate learning to speak,
listen, and read. Meanwhile, the delayed interaction
method is applied in learning writing as a follow-up
of reading lessons done by sending the material and
reading material by electronic mail to the learners.
Response to written readings is conveyed back to the
instructor via e-mail facility to the teacher.
Teachers in the BIPA e-learning program face
various dynamics while teaching and interacting
with BIPA learners online. As a solution, WB
always tries to take advantage of the cutting-edge
technological devices to deliver the Indonesian
language learning materials and media to learners
taking courses from overseas. It is also supported by
the teachers’ excellent performance that is always
adaptive to the development of information
technology so that interaction with learners can take
place smoothly and effectively.
4.4 Product Evaluation
The product evaluation referred to in this study is
more on the effort to identify the learning outcomes
that are aligned with the goals and needs formulated
at the beginning of the program. Evaluation is also
done by linking various information on context,
input, and process so as to know the quality,
usefulness, advantages, and significance of the
program to be further processed in order to
determine the sustainability of the e-learning
program.
From the teacher's perspective, it is revealed that
most learners require formal evaluation because they
are not bound to complete specific programs in their
institution of origin. The evaluation is limited to the
provision of feedback at the time of learning without
being supported by the documentation of the results
of a systematic evaluation. Direct evaluation results
are used to refine the learning process by the
teacher, such as updating the material, adding the
media, or providing corrections on the learners’
language problems.
From the perspective of the R & D unit, product
evaluation is done by gathering information from
learners about their level of satisfaction in joining
the e-learning program at WB. From the customer
survey forms given to the learners, 85% of learners
felt very satisfied with the program and they
suggested some ideas to improve the interactivity
between teachers and learners in terms of internet
connection and the diversity of interactions.
Evaluation of E-Learning in Indonesian as a Foreign Language
485

5 CONCLUSIONS AND
RECOMMENDATIONS
Planning, implementing, and evaluating an e-
learning program to teach Indonesian as a foreign
language can be a complex task because e-learning
involves multiple dimensions and aims to meet the
students’ needs and e-learning program
stakeholders. The eight dimensions of e-learning
should be synergized and integrated so that the
quality of e-learning environment can be seen in all
components and be able to support the learning
process effectively (Khan, 2005a; Khan, 2005b;
Khan, 2005c).
This study aimed to evaluate the effectiveness of
e-learning program to teach Indonesian as a foreign
language at WB Yogyakarta through the perspective
of instructors/teachers, curriculum designers, IT
staff, and R & D unit. The results of the study
indicated that the e-learning program to teach
Indonesian as a foreign language at WB Yogyakarta
needs to be improved in some dimensions of
context, input, process, and product. The findings
revealed that some improvements in curriculum
design, physical conditions, e-learning artefact, e-
learning objects, technology, interface design, and
evaluation dimensions of the e-learning program
were required to make the program more effective in
the future. Based on the findings and analysis above,
there are some recommendations that researchers
proposed to implement the ideal e-learning programs
for BIPA in WB.
1. The forms of new technology and e-learning
artefacts used in e-learning programs need to
be incorporated into e-learning platforms that
are specifically developed in accordance with
the characteristics and potential of the
institutions.
2. Curriculum designers for e-learning programs
need to redesign the e-learning environment by
integrating the aforementioned e-learning
dimensions.
3. Comprehensive and integrated program
evaluation is required so that the dimensions of
e-learning environments can be seen in the
quality and new designs are obtained for the
improvement and sustainability of the BIPA e-
learning programs.
REFERENCES
Azimi, H. M., and Rahmani, R., 2013. Importance of
Needs assessment of Implementation of E-learning in
College of Education. International Journal of
Information and Computation Technology, 3(5),
pp.377-382.
Ehler, U. D., and Hilera, J. R., 2012. Special Issue on
quality in e-learning. Journal of Computer Assisted
Learning, 28(1), pp.1-3.
Fitzpatrick, T., 2012. Key Success Factors of eLearning in
Education: A Professional Development Model to
Evaluate and Support eLearning. US-China Education
Review A, 9, pp.789-795.
Fitzpatrick, J. L., Sanders, J. R., and Worthen, B. R., 2012.
Program Evaluation: Alternative Approach and
Practical Guidelines. Upper Saddle River, NJ:
Pearson Education, Inc.
Hakan, K. and Seval, F., 2011. CIPP evaluation model
scale: development, reliability and validity. Procedia-
Social and Behavioral Sciences, 15, pp.592-599.
Joint Committee on Standards for Educational Evaluation,
1994. The Program Evaluation Standards. Newbury
Park, CA: Sage Publications.
Khan, B. H., 2005a. A Comprehensive E-learning Model,
Je-LKS. Journal of e-learning and Knowledge Society,
1(1), pp.33-44.
Khan, B. H., 2005b. Managing E-learning Strategies:
Design, Delivery, Implementation and Evaluation. IGI
Global.
Khan, B. H., 2005c. E-learning Quick Checklist. IGI
Global.
Lodico, M.G., Spaulding, D.T. and Voegtle, K.H.,
2010. Methods in educational research: From theory
to practice (Vol. 28). John Wiley & Sons.
Oluwatobi, O., 2015. Evaluation of Postgraduate
Programme of Babcock University using CIPP Model.
International Journal of Eduational Research and
Development, 4(1), pp. 005-012.
Owston, R., 2008. Models and Methods for Evaluation. In
J. Michael Spector (eds). Handbook of Research on
Educational Communications and Technology.
Lawrence Erlbaum Associates, New York.
Patton, M.Q., 2002. Qualitative Research and Evaluation
Methods. CA: Sage Publications, Inc.
Stufflebeam, D.L., 2003. The CIPP model for evaluation.
In International handbook of educational evaluation,
pp. 31-62.
Stufflebeam, D.L., Coryn, and Chris, L.S., 2014.
Evaluation Theory, Models, and Applications. San
Francisco: Jossey-Bass A Wiley Brand.
Tiantong, M. and Tongchin, P., 2013. A Multiple
intelligences supported web-based collaborative
learning model using Stufflebeam’s CIPP evaluation
model. International Journal of Humanities and Social
Science, 3(7), pp.157-165.
CONAPLIN and ICOLLITE 2017 - Tenth Conference on Applied Linguistics and the Second English Language Teaching and Technology
Conference in collaboration with the First International Conference on Language, Literature, Culture, and Education
486