Self-consistent Peer Ranking for Assessing Student Work - Dealing with Large Populations

Kees van Overveld, Tom Verhoeff

Abstract

Assessing large populations of students puts a serious burden on teaching staff capacity. For open-format assignments, automation of the reviewing process can offer only limited support. Peer ranking is a partial solution to the problem, with the added benefit that students' critical reading skills are developed. We see two remaining problems, however: (1)~for students, it is a major challenge to assign marks on an absolute scale, and (2)~students' competence in reviewing may vary significantly--- so not all peer reviews should have a similar weight in the process. To remedy these shortcomings, we suggest an approach to peer ranking, inspired by Jon Kleinberg's HITS-algorithm, where both the students' assignment results and the quality of their double anonymous peer reviews are algorithmically ranked. Based on preliminary model calculations, we estimate that this strategy may reduce the required effort for reviewing open-format assignments approximately by a factor of ten. A first large-scale pilot with this method will take place in undergraduate courses at Eindhoven University of Technology, spring 2013. Since this involves about 900 students, automated support is a must. We describe the peer reviewing facilities that were introduced in our web-based education support system named peach3.

References

  1. Allain, R., Abbot, D., and Deardorff, D. (2006). Using peer ranking to enhance student writing. Physics Education, 41(3):255-258.
  2. Golub, G. H. and Van Loan, C. F. (1996). Matrix Computations (3rd Ed.). JHU Press.
  3. Kleinberg, J. (1999). Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604-632.
  4. Lu, R. and Bol, L. (2007). A comparison of anonymous versus identifiable e-peer review on college student writing performance and the extent of critical feedback. J. of Interactive Online Learning, 6(2):100-115.
  5. Mellenbergh, G. J. (2011). A Conceptual Introduction to Psychometrics: Development, Analysis, and Application of Psychological and Educational Tests. Eleven International Publishing.
  6. Sadler, P. M. and Good, E. (2006). The impact of self- and peer-grading on student learning. Educational Assessment, 11(1):1-31.
  7. Scheffers, E. and Verhoeff, T. (Accessed Nov. 2012). peach3. http://peach3.nl/.
  8. van Zundert, M. J. (2012). Conditions of Peer Assessment for Complex Learning. PhD thesis, Maastricht University.
Download


Paper Citation


in Harvard Style

van Overveld K. and Verhoeff T. (2013). Self-consistent Peer Ranking for Assessing Student Work - Dealing with Large Populations . In Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU, ISBN 978-989-8565-53-2, pages 399-404. DOI: 10.5220/0004352903990404


in Bibtex Style

@conference{csedu13,
author={Kees van Overveld and Tom Verhoeff},
title={Self-consistent Peer Ranking for Assessing Student Work - Dealing with Large Populations},
booktitle={Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU,},
year={2013},
pages={399-404},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004352903990404},
isbn={978-989-8565-53-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU,
TI - Self-consistent Peer Ranking for Assessing Student Work - Dealing with Large Populations
SN - 978-989-8565-53-2
AU - van Overveld K.
AU - Verhoeff T.
PY - 2013
SP - 399
EP - 404
DO - 10.5220/0004352903990404