Smart Like a Fox: How Clever Students Trick Dumb Automated Programming Assignment Assessment Systems

Nane Kratzke

2019

Abstract

This case study reports on two first-semester programming courses with more than 190 students. Both courses made use of automated assessments. We observed how students trick these systems by analysing the version history of suspect submissions. By analysing more than 3300 submissions, we revealed four astonishingly simple tricks (overfitting, evasion) and cheat-patterns (redirection, and injection) that students used to trick automated programming assignment assessment systems (APAAS). Although not the main focus of this study, it discusses and proposes corresponding counter-measures where appropriate. Nevertheless, the primary intent of this paper is to raise problem awareness and to identify and systematise observable problem patterns in a more formal approach. The identified immaturity of existing APAAS solutions might have implications for courses that rely deeply on automation like MOOCs. Therefore, we conclude to look at APAAS solutions much more from a security point of view (code injection). Moreover, we identify the need to evolve existing unit testing frameworks into more evaluation-oriented teaching solutions that provide better trick and cheat detection capabilities and differentiated grading support.

Paper Citation

in Harvard Style

Kratzke N. (2019). Smart Like a Fox: How Clever Students Trick Dumb Automated Programming Assignment Assessment Systems.In Proceedings of the 11th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-367-4, pages 15-26. DOI: 10.5220/0007424800150026

in Bibtex Style

@conference{csedu19,
author={Nane Kratzke},
title={Smart Like a Fox: How Clever Students Trick Dumb Automated Programming Assignment Assessment Systems},
booktitle={Proceedings of the 11th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2019},
pages={15-26},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0007424800150026},
isbn={978-989-758-367-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 11th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Smart Like a Fox: How Clever Students Trick Dumb Automated Programming Assignment Assessment Systems
SN - 978-989-758-367-4
AU - Kratzke N.
PY - 2019
SP - 15
EP - 26
DO - 10.5220/0007424800150026