EFFECTIVE ONLINE TESTING

Nurturing the Digital Natives

Gabriele Frankl

, Peter Schartner

and Gerald Zebedin

Department of e-learning, Alpen-Adria-Universität Klagenfurt, Universitätsstrasse 65-67, 9020 Klagenfurt, Austria

System Security Group, Alpen-Adria-Universität Klagenfurt, Universitätsstrasse 65-67, 9020 Klagenfurt, Austria

Keywords: Secure Online Testing, Secure Exam Environment, Moodle, Safe Exam Browser, Survey.

Abstract: This paper aims to take a glance at a flexible solution for online-testing at the Alpen-Adria-Universität

Klagenfurt (AAUK). We will present a recently implemented software that allows students to conduct their

exams online and directly on their own devices, while at the same time preventing them accessing locally

stored files or non-specified internet pages. A survey among the participating students showed that the general

attitude of students is very positive. Especially among lecturers the demand for the secure exam environment

(SEE) on their courses has increased tremendously. For the future we plan to further develop the secure exam

environment and we intend to use larger lecture halls i.e. for synchronous online testing of approximately 250

students.

1 INTRODUCTION

The various problems connected with the testing of

knowledge are known: standardized tests run the risk

of only testing “superficial” knowledge, whereas

free-text or face-to-face tests are criticised for not

being objective enough (whether or not the teacher is

aware of this fact). In order to provide an efficient and

transparent process, more and more online tests are

applied in a multi-faceted way at the University of

Klagenfurt (AAUK). Currently, fully standardised

tests, mixed test forms with standardised and free-text

questions, solely free-test questions as well as tests

incorporating specific software tools e.g. spread sheet

programs and online tests for writing literary essays,

are in use.

In general, other advantages are connected with

online-exams: they are much more efficient to carry

out, create automated feedback to standardized

questions and facilitate the readability of free-text

answers. Also the reusability of questions gives back

time resources that can be used for areas of teaching

requiring face-to-face contact. Furthermore,

Austrian students usually do not make use of their

right to inspect the results of their exams by

personally contacting the lecturer and therefore they

do not receive potentially valuable feedback.

However, if the feedback is available online, our

experience shows that nearly all students use the

possibility to inspect their exam online (see Frankl,

Scharter and Zebedin 2011a).

Using up-to-date teaching methods is an essential

concern of our university and contemporary teaching

requires also up-to-date testing and examining

methods. Therefore, the e-learning department of the

AAUK developed the “secure exam environment”

(SEE) in cooperation with the System Security Group

of the AAUK.

2 LARGE SCALE ONLINE

TESTING

One possibility for conducting online exams is in

computer rooms. However, they are in most cases

much too small. Where there are large enough

computer rooms, these are usually very costly and

therefore not economically viable. Additionally,

maintenance of these rooms is necessary, single

computers cannot work and it is also very laborious

to install the specially required software tools.

Hence, the SEE is based on the usage of existing

students’ resources. An evident advantage is the

flexibility of the SEE, as it can be used on a grand

scale. Another advantage for the students is that they

can use their own devices, which they are used to,

which

is probably a benefit especially in an exam

340

Frankl G., Schartner P. and Gerald Z..

EFFECTIVE ONLINE TESTING - Nurturing the Digital Natives.

DOI: 10.5220/0003920703400343

In Proceedings of the 4th International Conference on Computer Supported Education (CSEDU-2012), pages 340-343

ISBN: 978-989-8565-06-8

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

situation where they may be stressed and nervous.

Certainly, when using the technical resources of

the students, loan devices are offered in case of

technical problems or for those students who do not

own a mobile device.

Nevertheless, despite the obvious advantages

there is one major pitfall with using student’s

devices, namely that students have access to their

local files, to the internet and therefore also to

communication tools. Thus, we developed the

“Secure Exam Environment” (SEE).

3 THE SECURE EXAM

ENVIRONMENT (SEE)

Without a doubt, online applications and exams both

come with massive security issues (see Weippl 2005,

Bartram and Hambleton 2008, and Scheuermann and

Björnsson 2009, pp. 84). However, the secure exam

environment can hinder the issue of cheating.

3.1 Security

Since KNOPPIX is said to be the Linux-distribution

with the broadest hardware support, a knoppix boot

system (www.knopper.net/knoppix/index-en.html)

was developed to meet the requirement that there

would be no access to students’ own files and data

as well as to Internet addresses and links. The

students boot the SEE from a flash drive or a DVD.

Corresponding firewall rules restrict the access to

external (online) resources. The only internet

resource that can be accessed from the SEE is the

server of the e-learning-platform Moodle

(http://moodle.com). As Moodle also provides

chatting capabilities and (in most cases) course

related material, we integrated the Safe Exam

Browser (SEB – see www.safeexambrowser.org/

news_en.html for details), an open source project

supported by the University Giessen and the ETH

Zurich, into Moodle-core 1.9.4. The SEB restricts

the access to the exam page only. Since the SEB is

only available for Windows XP, Windows 7 and

MAC OS X, we have to run a minimized

WindowsXP that provides exactly the necessary

programs in a virtual machine (www.virtualbox.org)

on the KNOPPIX system (see Fig. 1 for a sketch of

the system). (see Frankl, Scharter and Zebedin

2011a and b).

After booting from USB or DVD, KNOPPIX

automatically starts the WindowsXP-Virtual

Machine,

then Windows XP automatically starts the

Figure 1: The operating principle of SEE (see Frankl,

Schartner and Zebedin 2011a).

SEB that automatically connects to the homepage of

Moodle. The students login into Moodle and select

the appropriate course and access the exam.

In order to make it work, the respective browser

security settings have to be chosen when setting up

the Moodle test. To guarantee that only students who

are present in person can access the online-exam, the

network address of the lecture hall has to be entered

in the designated field in the quiz security settings.

3.2 Safety and Privacy

Electronic exam environments massively depend on

the availability of (information) technology. During

an online exam our system relies on the online

connection between the SEB and the Moodle Server.

In order to provide the maximum availability of

the network connection we only use cat5-wired LAN

at this time. However, LAN also doesn’t guarantee a

100 % stable connection.

Hardware failures of laptops are not such a

problem, since students can continue the exam on one

of our loan-devices.

The answers provided by the students and the

grades achieved relate specifically to individual

students and hence according to the Austrian Data

protection act (see Republic of Austria 2009) their

privacy has to be protected. We use encrypted and

authenticated transmission lines between the SEB

and the Moodle server (HTTPS), and the Moodle

server uses login/password-authentication for the

students and the lecturers and grants access to

authorized persons only. Hence, we can guarantee the

privacy of exam data (see Frankl, Scharter and

Zebedin 2011a and b)

3.3 Organisational Aspects

During our first testruns we encountered network

timeout errors. After debugging we discovered a

misconfiguration in the DHCP settings assigned to

the lecture hall used for exams. The DHCP lifetime

was set to 120 minutes, which led to network

EFFECTIVEONLINETESTING-NurturingtheDigitalNatives

341

timeouts and connection losses between the SEB and

the moodle server. Such organisational aspects have

to be mentioned.

Furthermore, the lecturers are supported by

e-tutors of the e-learning department of the university

to properly set up their online-exams. Moreover the

students have to be informed in due time before the

exam about the action chain for the online-exam. The

e-learning department of the AAUK offers checklists

for students and lecturers.

The implementation and execution of online

exams requires a certain amount of time. Therefore,

an efficient and well-planned organisation is vital.

4 STUDENTS’ FEEDBACK

By the end of February 2012, we had conducted 40

online exams with the SEE. The online tests were a

mix of pure single or multiple choice questions, other

standardised question types, free-text answers, or – in

one case - even group work using explicitly allowed

materials.

The courses come from the area of business

administration, information science, mathematics,

geography, media and communication science,

English and American studies, German language and

literature studies as well as from educational studies.

Students, who used the SEE for their final exams,

were asked to give feedback after the online-exam in

their respective Moodle course.

In total 965 students participated in the

online-exams. 274 students (28,4 %) submitted their

feedback.

The students were asked for their assessment of

the online exam and given the following options for

their answers: very negative / negative / positive /

very positive. 26 % of the students evaluated online

exams in general as “very positive”, 56.8 % as

“positive, 15 % as “negative” and 2.2 % of the

students considered as “very negative” (see Fig. 2).

Figure 2: Students' attitude about online exams.

Students were also asked about advantages and

disadvantages of online exams and provided a variety

of wide ranging answers to these open questions.

Fig. 3 shows the main advantages from the students’

point of view.

Figure 3: Advantages of online exams from the students’

point of view.

The most frequently mentioned disadvantages -

from a student perspective – are summarised in Fig.

Figure 4: Disadvantages of online exams from the students’

point of view.

5 ON-GOING PROBLEMS AND

FUTURE WORK

The demand from lecturers to use this method of

online-examination is extremely high. Due to the

various requirements of the lecturers, the present

Moodle-question types and multiple-choice

evaluation options do not exactly meet all the needs

of our lecturers. There is room for improvement to

enlarge the testing-possibilities.

Additionally, it has to be highlighted that the

requirements for the SEE are an electrical socket and

2,2%

15,0%

56,8%

26,0%

0,0% 10,0% 20,0% 30,0% 40,0% 50,0% 60,0%

very negativ

negativ

positiv

very positiv

0 20406080

environmentally friendly

much more convenient,

interesting & better

no difference

Improved structure,

readability & correction …

saves time

fast results

0 20406080

simply prefer paper &

…

longer preparation time

problems with (types

…

lack of structure and

…

more time required

technical issues

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

342

network access at each and every seat in the lecture

hall. At the moment, there are only a limited number

of lecture halls that offer these prerequisites. Hence

restructuring work will take place in order to install

the necessary technical equipment in the larger

lecture halls (for synchronous online testing of

approximately 250 students). Furthermore, the SEE

will be adapted to the new performance requirements.

In addition, a flexible registration system will be

developed. Students will not in the future be

dependent anymore on a fixed exam date, but can

conduct their exam within a stipulated testing period.

The online exam will then be generated out of a large

enough question pool in real-time for each and every

student.

6 OUTLOOK

In many disciplines traditional methods for

performance assessment are, in terms of didactics

completely inappropriate. However, the classical

testing is in many cases much better than its

reputation. Sometimes it is necessary to study hard

facts and to test these. In these situations classical

knowledge appraisals are appropriate. However,

some guidelines on creating useful and high quality

multiple-choice questions were developed by August

Fenk from the AAUK and is available for lecturers.

If classical exam methods are managed and

supported electronically, time is saved for those areas

of teaching that require face-to-face contact, a lot of

time and that currently students miss out on, such as

in-depth discussions, reflections, oral exams, or if

applicable, extra coaching in these areas.

REFERENCES

Bartram D., Hambleton, R. K., 2008. Computer-Based

Testing and the Internet: Issues and Advances, John

Wiley & Sons.

Frankl, G., Schartner, P., Zebedin, G., 2011a. The „Secure

Exam Environment“ for Online Testing. In

Proceedings of the World Conference on E-Learning in

Corporate, Government, Healthcare, and Higher

Education (ELEARN) 2011, pp. 1201-1211.

Chesapeake, VA:AACE.

Frankl, G., Schartner, P., Zebedin, G., 2011b. Secure

Online Exam Environment. Submitted at the EDUCON

2011.

Republic of Austria, 2009. Datenschutzgesetz 2000

(DSG2000), BGBl. I Nr. 165/1999, Online (last visit

26.8.2011), http://www.ris.bka.gv.at/

GeltendeFassung.wxe?Abfrage=Bundesnormen&Gese

tzesnummer=10001597 http:// www.dataprotection.eu/

pmwiki/pmwiki.php?n=Main.AT

Republic of Austria, 2011. Universitätsgesetz 2000, BGBl.

I Nr. 120/2002, Online (last visit 26.8.2011) http://

www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bu

ndesnormen&Gesetzesnummer=20002128

Scheuermann, F., Julius Björnsson, J., 2009. The Transition

to Computer-Based Assessment, (last visit 26.8.2011)

http://www.gesci.org/assets/files/reporttransition.pdf

SoftwareSecure, 2011. Securexam. Online (last visit

26.8.2011), http://www.softwaresecure.com

Weippl, E.R., 2005. Security in E-Learning, Advances in

Information Security, Springer.

EFFECTIVEONLINETESTING-NurturingtheDigitalNatives

343