EXPERIENCES WITH WEB-BASED PEER ASSESSMENT OF

COURSEWORK

Hans H¨uttel and Kurt Nørmark

Department of Computer Science, Aalborg University, Selma Lagerl¨ofs Vej 300, Aalborg, Denmark

Keywords:

Peer Assessment.

Abstract:

We describe experiments with a web-based system for peer assessment in a course on automata theory and

program semantics. The paper describes the web-based assessment system, and experiences from the ﬁrst

round of use in 2011. We ﬁnd a correlation between a high level of peer reviewing activity and a high grade

obtained at written exam.

1 INTRODUCTION

In this paper we describe an experimental system

whose aim is to address some of the challenges that

we have found when giving undergraduate courses.

1.1 Challenges to University Teaching

The proportion of school leavers that enter higher ed-

ucation is steadily increasing, and this has led to the

emergence of a larger group of students with different

attitudes towards learning.

Biggs and Tang (Biggs and Tang, 2007) distin-

guish between the dedicated student –‘Susan’ – and

the less dedicated student – ‘Robert’. ‘Susan’ is inter-

ested in her subject and what she learns is important

to her, while ‘Robert’ is not so interested in his sub-

ject, uses notably less effort and focuses on how to

be able to qualify for a job. Traditionally, university

teaching has focused exclusively on ‘Susan’; an im-

portant challenge has been to cater also to the needs

of ‘Robert’ while keeping the focus on the needs of

the subject and of learning.

Observations suggests that whatever has been

taught in a programme tends to be forgotten by stu-

dents relatively soon. A survey by Anderson et al.

(Anderson et al., 1998) documents that a majority of

undergraduate-level mathematics were able to recall

only little or none of what was taught in their ﬁrst-

year courses.

Our own experience is similar. We come across

students at masters level asking us for help because

they need to brush up their knowledge of central top-

ics from courses that they passed earlier.

Another challenge is that, because of the increas-

ing student intake, lecturers need to spend more time

evaluating students but will not have more resources

at their disposal.

Lauv˚as and Jakobsen (Lauv˚as and Jakobsen,

2002) point out both of these problems and argue that

it is due to the fact that ‘Robert’ will choose an instru-

mentalist approach to learning whose primary goal is

that of being able to pass the exam. ‘Robert’ will not

prepare for course sessions but tries to cram the mate-

rial during the last few days before the exam. This is

a hindrance to deep learning.

The challenge is then to teach such that some

‘Robert’ students will end up becoming ‘Susans’ and

in such a way that learning is improved, that deep

learning is encouraged and in such a way that the

teacher does not use more resources in this process.

1.2 Formative Evaluation through Peer

Assessment

Peer assessment is a strategy that has received a great

deal attention over the past 15 years in the world of

education research. For a recent overview, see the

survey paper by Topping (Topping, 2009). It has of-

ten often been argued that peer assessment can im-

prove the reﬂective processes of learning, since stu-

dents through the assessment process will have to re-

ﬂect on how their fellow students are approaching the

task of learning.

One often distinguishes between

• summative evaluation, which is the assessment

whose aim is to determine if the learning goals

113

Hüttel H. and Nørmark K..

EXPERIENCES WITH WEB-BASED PEER ASSESSMENT OF COURSEWORK.

DOI: 10.5220/0003918401130118

In Proceedings of the 4th International Conference on Computer Supported Education (CSEDU-2012), pages 113-118

ISBN: 978-989-8565-07-5

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

of a teaching activity have been reached, and

• formative evaluation, which is the assessment

whose aim is to guide the students so they will

know how their learning is proceeding and what

they should do to improve it. It is through forma-

tive evaluation that reﬂection can happen.

Often summative evaluation will take the form of

a formal examination. On the other hand, formative

evaluation will be part of the learning process. One

can therefore seek to incorporatea notion of formative

evaluation as part of teaching activities, and this is

where peer assessment comes in.

The claims that Lauv˚as and Jakobsen make are

(Lauv˚as and Jakobsen, 2002): If formative evaluation

is thorough and if one insists that the students make a

substantial effort throughout the course, the students

will learn much more and will remember the material

much better and for a longer period of time. The exam

itself will become a different experience for students,

since they will be examined on the basis of material

with which they are already well acquainted and feel

that they master. Lauv˚as uses the analogy of athletes

that practice regularly and are told by their coach how

to improve their performance.

Lauv˚as and Maugesten have experimented with

methods for restructuring teaching via peer assess-

ment (Maugesten and Lauv˚as, 2004). The activities

have been laid out such that all students would explic-

itly need to adopt a ‘Susan’-style learning strategy.

1.3 Existing Solutions

There has already been a fair amount of work on au-

tomating the administrative aspect of the peer assess-

ment strategy.

Michael de Raadt has created a peer review mod-

ule (de Raadt, ) that can be used with Moodle (Moo-

dle, ). One limitation of this module is that peer

reviews take the form of a series of multiple-choice

questions for which boxes are to be ticked. Other

forms of assessment are not possible.

In a series of papers, Joy and Sitthiworachart have

focused on how to use web-based peer assessment to

assist deep learning in programming classes (Sitthi-

worachart and Joy, 2004; Ward et al., 2004). In this

work, they have among other things, focused on ask-

ing the students to comment on speciﬁc aspects of so-

lutions to programming exercises. Their results indi-

cate that web-based peer assessment can indeed assist

in the process of deep learning.

1.4 Our Setting

We have developed and used an experimental system

for peer assessment used in the course Syntax and Se-

mantics taught at our university. Our goal has been

to gain experience in how to increase student efforts

and improve learning through web-based methods for

peer assessment. We have based the design of our

system on previous, largely paper-based strategies for

peer assessment.

2 PREVIOUS EXPERIENCE

From 2004 to 2007, the ﬁrst author has experimented

with methods of peer assessment in another theory

course on computability and complexity theory.

A questionnaire-based survey has shown that, in

the past, a student would spend only 30 minutes

preparing for a course session. While peer assess-

ment dramatically increased student efforts, the solu-

tion was entirely paper-based and this required a sig-

niﬁcant effort of all involved. The incentive for stu-

dents was that the participating students would have a

reduced syllabus at the oral exam.

The assessment expected of students involved that

of reviewing the solutions to textbook-style problems

solved by their fellow students. Here, the problem

turned out to be one of greatly varying levels of un-

derstanding: students that had difﬁculting solving a

problem were often unable to provide meaningful re-

views. Quite often, our conversations with students

during commenting sessions revealed that the student

had not read the relevant pages of the course textbook.

Finally, some ‘Robert’-students very consistently

handed in ‘non-solutions’ in which they simply wrote

that they could not solve the problems that had been

stated. These same students would also write very

brief reviews of solutions. As they had handed in so-

lutions and participated in the commenting process

and thereby followed the formal procedures, it was

hard to argue that they should not have a reduced

exam syllabus even if they had done little actual work.

3 THE CURRENT COURSE

The course Syntax and Semantics is found at the

4th semester of the undergraduate programmes in

Computer Science and in Software Technology. The

course covers formal language theory and program-

ming language theory.

The course consists of 15 4-hour sessions, divided

into 15 practical sessions of 2 hours each and a 90

minute lecture. At the same semester, there were three

other courses of 3 ECTS each and an 18 ECTS stu-

dent project. As the students collaborate on projects

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

114

in groups of 5 to 7 students and are assigned a group

room, it is common practice for them to also work on

course practicals in these project groups.

For the Computer Science students, the exam in

Syntax and Semantics is a 3-hour written exam. Stu-

dents following the Software Technology programme

did not participate in the written exam. These stu-

dent had the course assessed as part of their semester

project which involves several other topics not related

to the course syllabus.

4 THE APPROACH

The course consisted of a variety of teaching activi-

ties, includes lectures and problem solving. However,

in this section we concentrate on the new develop-

ment, namely the approach that we have followed for

peer assessment, and how it led to the design of our

system. First, the overall approach to questioning, an-

swering, and reviewing is described. This includes the

means of incentives provided by the approach. Next,

the workﬂow of a single session - called an answering

session - is explained.

4.1 The Peer Assessment Approach

Our peer assessment strategy has been based on the

following principles.

• Students should read the course text and reﬂect on

its content, so the peer assessment exercises asked

a series of questions directly related to the text.

• Each student who answers the question becomes a

reviewer of another student. The student reviewer

should as their peer amend and correct the an-

swers to the text-related questions instead of try-

ing to ﬁgure out how the answer is incorrect.

• Students should be encouraged to participate. The

incentive was that their answers to text questions

would be available for them to use at the written

exam. No other textual aids would be allowed at

the exam. The collection of answers was kept in

the form of a portfolio for each student. The exam

questions were the same for all students, regard-

less of their participation in peer assessment or

lack thereof.

• Students should use L

X for their answers to

text-related questions, as this is a system that they

are familiar with is well-suited for technical writ-

ing. A further advantage has been that the web-

based implementation could easily handle the text

ﬁles used by L

X. To this end, the students

would be supplied with a standard preamble that

they must use.

Some examples of text-related questions (trans-

lated from Danish) can be found in Figure 1.

(From the session about regular expressions) Let

R be a regular expression. Is it always the case

that the regular expression R∪ ε denotes the same

language as R? If the answer is yes, then explain

why. If the answer is no, then providea counterex-

ample.

(From the session about scope rules described us-

ing structural operational semantics) Here is a

big-step transition rule. Explain which kind of

scope rules are captured by this particular transi-

tion rule.

(CALL-1)

env

, env

′

⊢ hS, stoi → sto

′

env

, env

⊢ h

call

p, stoi → sto

′

where env

p = (S, env

′

)

Figure 1: Examples of text-related questions used (trans-

lated from Danish).

4.2 The Workﬂow of an Answering

Session

Each of the 15 answering sessions of the course had

the following workﬂow:

1. After the lecture, the teacher would publish a

X ﬁle Doc

with questions about the associ-

ated part of the text to the course webpage.

2. No later than by the morning of the session, at

deadline d

, the student would then answer the

questions found in Doc

and hand the answers in

the form of Doc

, an amended version of Doc

3. Just after the deadline d

the Doc

ﬁles would be

distributed among the participating students for

commeting and reviewing. Each reviewer com-

piles Doc

, read it and amend/modify the answers.

This would then lead to a new version of the L

ﬁle, Doc

4. The students would ﬁnally get back Doc

and be

able to obtain it and either use it as their own ﬁ-

nal version or revert to the one that they originally

wrote. The ﬁnal outcome of the session is then

a ﬁle Doc

which is added to the portfolio of the

student.

This workﬂow is summarized in Figure 2.

EXPERIENCESWITHWEB-BASEDPEERASSESSMENTOFCOURSEWORK

115

Figure 2: The workﬂow supported by the system, showing the documents that ﬂow between the teacher, author, and reviewer

together with the deadlines that apply.

4.3 The Implementation

The role of the system was to support the workﬂow

described above, and make it possible to manage the

portfolio of each student in an easy manner.

The system allows each student to log into a sta-

tus page for each lecture in the course (with use of a

valid username and password). Until the given dead-

line d

the status page allows upload of the L

X ﬁle

Doc

. After the deadline the system appoints a re-

viewer for each student that has submitted a Doc

the current round of questions. The allocation of re-

viewers is done by computing a permutation of the

students that have submitted a Doc

for this session.

If possible, the permutation is formed in such a way

that the reviewer assigned to a given student is in an-

other project group than that of the original author.

With the reviewers appointed, the status page of

the student S is augmented with information about

the reviewer of S, R(S), and the student that will be

reviewed by S. In addition, R(S) gets access to the

material he or she is supposed to review via his or her

status page together with a new deadline d

. When a

revised Doc

is submitted by R(S), the original author

S receives an email from the system, and the review is

made available at the status page for S. If or when a

review is received, S can submit a ﬁnal version Doc

which allows S to elaborate the original contribution

and/or the changes made by the reviewer R(S).

The contributionto the portfolio of student S in the

current round becomes Doc

if it exists, else Doc

if it

exists, else Doc

if it exists, else the empty document.

The system provides the course teacher with an

overview page, in terms of a table that shows the sta-

tus of all students for every answering session so far.

The teacher can activate the appointment of reviewers

(construction of the permutation mentioned above)

from the overview page. In addition, the teacher

has access to the status page of each student via the

overview page, and the individual L

X documents

submitted by authors and reviewers.

At any point in time, it is possible for a student

to construct an aggregated portfolio of answers to all

questions. Behind the scenes the web system pro-

cesses a L

X ﬁle, which is aggregated by the rele-

vant pieces submitted by the author and the reviewers

of the author. If there are problems with the process-

ing (due to errors in the L

X document), the student

is supposed to deal with these via the L

X log ﬁle,

which also is made available.

The administrative system setup was mainly done

programmatically, via Lisp text ﬁles. The system

needs information about all participating students

(full name, email address, encrypted password, and

group number id). In addition, the system needs in-

formation about deadlines for each lecture, as illus-

trated in Figure 2). The system relies on a list of all

deadlines (relative to Figure 2). The other dead-

lines are calculated by use of ﬁxed offsets. In a future

version of the system it will be attractive to have a

web-interface to both student administration and the

temporal setup of deadlines.

The current system is implemented (by the second

author) in R5RS Scheme (Kelsey et al., 1998) with

use of LAML (Nørmark, 2005) for CGI programming

purposes. Administrative data are stored in ﬁles - one

ﬁle per student per answering session.

5 EXPERIENCES

In this section we assess the experiences that we have

had using the system in our teaching.

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

116

No answer at all 136

Only ﬁrst answer 413

First answer and review 213

First and second answer, no review 67

First answer, review, and second answer 164

Only second answer 27

Total 1020

Figure 3: Distribution of the answering sessions.

5.1 A Quantitative Analysis

We have performed a quantitative analysis of data by

using the information in the implemented web system

juxtaposed with grades obtained by the students that

attended the written exam. 68 students participated in

15 rounds of answering questions, leading to a total

of 1020 answering sessions each of which follows the

workﬂow shown in Figure 2. Figure 3 shows a classi-

ﬁcation of the 1020 potential answering sessions.

Of the 68 students, 26 were Computer Science

students, and as such they participated in the written

exam, where they got a grade from the Danish 7-step

scale.

Figure 4 lists the number of answers (rang-

ing from 0 to 15) and the corresponding number of

students who submitted that particular number of an-

swers. In addition, we show the list of grades obtained

by the computer science students who were required

take the exam. Thus, for instance, 15 students sub-

mitted 14 answers, and among these, 6 participated in

the exam where they obtained the grades -3, -3, 4, 4,

7, and 10, respectively. One immediately notices that

the majority of the students, who attended the writ-

ten exam, submitted answers in almost all of the 15

rounds. This is less pronounced for software students,

who did not participate in the exam.

The activity of each student can be measured in

terms of the number of answers provided in total (0

.. 15), or more ﬁne-grained, using the total number of

answers (0, 1, or 2) and reviews (0 or 1) submitted for

each answering session.

On the other hand, Figure 3 reveals that as many

as 57 % of the answers provided were not followed

by a review. This indicates that the aspect of peer as-

sessment was not seen by the students as a necessary

component of the teaching activity.

5.2 A Qualitative Analysis

By using the system and interacting with the students,

we have made some observations of a more qualita-

The Danish 7-step grading scale is compatible with the

ECTS scale: 12 = A, 10 = B, 7 = C, 4 = D, 02 = E, 00 = Fx,

-3 = F. The grade 02 is the lowest passing grade.

Number of

answers

Number of

students

List of grades obtained

(Computer Science

students only)

0 1 00

1 0 none

2 1 none

3 0 none

4 0 none

5 1 none

6 0 none

7 3 00

8 1 none

9 2 none

10 2 4

11 3 7

12 4 none

13 3 none

14 15 -3 -3 4 4 7 10

15 32 00 00 02 4 4 4 7 7 7 7 7

10 10 10 10 12

Figure 4: Exam results.

tive nature.

Unfortunately, it turned out that again, some stu-

dents spent more effort trying to circumvent the sys-

tem than in engaging in the learning process. As al-

ready noticed, some students uploaded answers but

did not comment on the answers provided by oth-

ers. This caused a considerable amount of frustra-

tion. Moreover, a couple of students discovered that

the implementation accidentally allowed them to sub-

mit Doc

without submitting Doc

. (This is the cause

of the row ‘Only second answer’ in Figure 3). This

meant that they could avoid the ﬁrst deadline and also

manage to avoid being involved in the commenting

round.

Other evidence suggests that students were not al-

ways careful in producing their answers. Before the

exam, all portfolios were generated and compiled by

the system. In some cases, the students had system-

atically modiﬁed the standard preamble, making it in-

compatible with the script for generating portfolios.

In several other cases, the L

X system was unable

to generate a portfolio due to simple L

X errors that

suggest that the author had never bothered to compile

EXPERIENCESWITHWEB-BASEDPEERASSESSMENTOFCOURSEWORK

117

the L

X ﬁle before submitting it.

It turned out that strict adherence to all deadlines

was difﬁcult to accomplish. Illness and other unfore-

seen circumstances led to introduction of a period of

time, where students could submit missing contribu-

tions. It turned out that 8% of all contributions were

received during that period. 92% of the contributions

were received on schedule, relative to the deadlines of

the answering sessions.

In feedback from some of the students it is pointed

out that the many deadlines that had to be met, were

a considerable burden. The work hereby enforced on

the students took time and efforts away from other

courses, and from the group project. In addition, it

was pointed out that the students wanted to be able to

revise their contributions, even after the last deadline

of an answering sessions. All taken together, it is the

impression of the students thatthe approach taken was

demanding.

6 CONCLUSIONS AND IDEAS

FOR FURTHER WORK

We have described a web-based system for peer as-

sessment. Our analysis of the exam results indicate

that there is a statistically signiﬁcant correlation be-

tween activity in the course and the result obtained at

the examination.

The connection between coursework and the exam

that follows is a powerful incentive for the students

to answer the questions and to aggregate a portfolio

that helps them during the exam. On the other hand,

the fact that more than half of the answers submitted

were not reviewed indicate that an important aspect

of the approach, namely that of peer assessment, did

not work out as planned. In order to improve the mo-

tivation for participating in the peer review work, we

consider to change the rules in the next round of use.

Our suggestion is is that a student can only have a

set of answers to text questions added to the portfolio

if the student has also submitted a review within the

same session.

Developing the administrative aspect of our sys-

tem is a topic of further work. Our system was sim-

ple in that respect; unforeseen events at student level

were dealt with on an ad hoc basis. More importantly,

the submitted ﬁles were never checked for L

X er-

rors. In a future version of the system, the day-to-

day administrative tasks and L

X issues should be

supported directly by the web-based system. An ap-

proach to this is to integrate our ideas into an exist-

ing web-based system for handling teaching activi-

ties. As Aalborg University is currently in the process

of switching its web-based course administration in-

terface to Moodle (Moodle, ), the ideas of our system

will be integrated into a plugin for Moodle.

REFERENCES

Anderson, J., Austin, K., Barnard, T., and Jagger, J. (1998).

Do third-year mathematics undergraduates know what

they are supposed to know? International Journal of

Mathematical Education in Science and Technology,

29(3):401–420.

Biggs, J. and Tang, C. (2007). Teaching for Quality Learn-

ing at University (Society for Research Into Highter

Education). Open University Press, 3 edition.

de Raadt, M. Michael de Raadt’s Peer Review Assign-

ment Type. http://www.sci.usq.edu.au/staff/deraadt/

peerReview.html.

Kelsey, R., Clinger, W., and Rees, J. (1998). Revised

report

on the algorithmic language Scheme. Higher-Order

and Symbolic Computation, 11(1):7–105.

Lauv˚as, P. and Jakobsen, A. (2002). Exit eksamen – eller?

Cappelen. In Norwegian.

Maugesten, M. and Lauv˚as, P. (2004). Bedre læring

av matematikk ved enkle midler? Rapport fra et

utviklingsprosjekt. Technical Report 6, Høgskolen i

Østfold. (in Norwegian).

Moodle. Moodle. http://moodle.org.

Nørmark, K. (2005). Web programming in Scheme with

LAML. J. of Funct. Prog., 15(1):53–65.

Sitthiworachart, J. and Joy, M. (2004). Effective peer as-

sessment for learning computer programming. In 9th

Annual Conference on the Innovation and Technology

in Computer Science Education (ITiCSE 2004), pages

122–126.

Topping, K. J. (2009). Peer assessment. Theory Into Prac-

tice, 48(1):20–27.

Ward, A., Sitthiworachart, J., and Joy, M. (2004). As-

pects of web-based peer assessment systems for teach-

ing and learning computer programming. In 3rd

IASTED International Conference on Web-based Ed-

ucation (WBE 2004), pages 292–297.

CSEDU2012-4thInternationalConferenceonComputerSupportedEducation

118