Better Feedback for Educational Online Judges

Anaga Mani, Divya Venkataramani, Jordi Petit, Salvador Roura

2014

Abstract

The verdicts of most online programming judges are, essentially, binary: the submitted codes are either “good enough” or not. Whilst this policy is appropriate for competitive or recruitment platforms, it can hinder the adoption of online judges on educative settings, where it could be adequate to provide better feedback to a student (or instructor) that has submitted a wrong code. An obvious option would be to just show him or her an instance where the code fails. However, that particular instance could be not very significant, and so could induce unreflectively patching the code. The approach considered in this paper is to data mine all the past incorrect submissions by all the users of the judge, so to extract a small subset of private test cases that may be relevant to most future users. Our solution is based on parsing the test files, building a bipartite graph, and solving a Set Cover problem by means of Integer Linear Programming. We have tested our solution with a hundred problems in Jutge.org. Those experiments suggest that our approach is general, efficient, and provides high quality results.

References

  1. Alon, N., Moshkovitz, D., and Safra, S. (2006). Algorithmic construction of sets for k-restrictions. ACM Trans. Algorithms, 2(2):153-177.
  2. Cheang, B., Kurnia, A., Lim, A., and Oon, W.-C. (2003). On automated grading of programming assignments in an academic institution. Computers & Education, 41(2):121-131.
  3. Chvatal, V. (1979). A greedy heuristic for the setcovering problem. Mathematics of Operations Research, 4(3):233-235.
  4. Douce, C., Livingstone, D., and Orwell, J. (2005). Automatic test-based assessment of programming: A review. ACM Journal on Educational Resources in Computing, 5(3).
  5. Foris?ek, M. (2006). Security of Programming Contest Systems. In Dagiene, V. and Mittermeir, R., editors, Information Technologies at School, pages 553-563.
  6. Giménez, O., Petit, J., and Roura, S. (2009). A pure problem-oriented approach for a CS1 course. In Hermann, C., Lauer, T., Ottmann, T., and Welte, M., editors, Proc. of the Informatics Education Europe IV, pages 185-192.
  7. Gurobi (2013). Gurobi Optimizer Reference Manual. http://www.gurobi.com.
  8. Ihantola, P., Ahoniemi, T., Karavirta, V., and Seppälä, O. (2010). Review of recent systems for automatic assessment of programming assignments. In Procs. of the 10th Koli Calling International Conference on Computing Education Research, pages 86-93. ACM.
  9. Joy, M., Griffiths, N., and Boyatt, R. (2005). The BOSS online submission and assessment system. ACM Journal on Educational Resources in Computing, 5(3).
  10. Karp, R. M. (1972). Reducibility Among Combinatorial Problems. In Miller, R. E. and Thatcher, J. W., editors, Complexity of Computer Computations, pages 85-103.
  11. Kurnia, A., Lim, A., and Cheang, B. (2001). Online judge. Computers & Education, pages 299-315.
  12. Leal, J. P. and Silva, F. (2003). Mooshak: a web-based multi-site programming contest system. Software: Practice and Experience, 33(6):567-581.
  13. Petit, J., Giménez, O., and Roura, S. (2012). Jutge.org: an educational programming judge. In Procs. of the 43rd ACM technical symposium on Computer Science Education, SIGCSE 7812, pages 445-450. ACM.
  14. Singh, R., Gulwani, S., and Solar-Lezama, A. (2013). Automated feedback generation for introductory programming assignments. In Boehm, H.-J. and Flanagan, C., editors, PLDI, pages 15-26. ACM.
  15. Tonin, N., Zanin, F., and Bez, J. (2012). Enhancing traditional algorithms classes using URI online judge. In 2012 International Conference on e-Learning and eTechnologies in Education, pages 110-113.
  16. Verdú, E., Regueras, L. M., Verdú, M. J., Leal, J. P., de Castro, J. P., and Queirós, R. (2012). A distributed system for learning programming on-line. Computers & Education, 58(1):1 - 10.
Download


Paper Citation


in Harvard Style

Mani A., Venkataramani D., Petit J. and Roura S. (2014). Better Feedback for Educational Online Judges . In Proceedings of the 6th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-021-5, pages 176-183. DOI: 10.5220/0004842801760183


in Bibtex Style

@conference{csedu14,
author={Anaga Mani and Divya Venkataramani and Jordi Petit and Salvador Roura},
title={Better Feedback for Educational Online Judges},
booktitle={Proceedings of the 6th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2014},
pages={176-183},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004842801760183},
isbn={978-989-758-021-5},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 6th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Better Feedback for Educational Online Judges
SN - 978-989-758-021-5
AU - Mani A.
AU - Venkataramani D.
AU - Petit J.
AU - Roura S.
PY - 2014
SP - 176
EP - 183
DO - 10.5220/0004842801760183