Ricardo Queirós, José Paulo Leal


Learning computer programming requires solving programming exercises. In computer programming courses teachers need to assess and give feedback to a large number of exercises. These tasks are time consuming and error-prone since there are many aspects relating to good programming that should be considered. In this context automatic assessment tools can play an important role helping teachers in grading tasks as well to assist students with automatic feedback. In spite of its usefulness, these tools lack integration mechanisms with other eLearning systems such as Learning Management Systems, Learning Objects Repositories or Integrated Development Environments. In this paper we provide a survey on programming evaluation systems. The survey gathers information on interoperability features of these systems, categorizing and comparing them regarding content and communication standardization. This work may prove useful to instructors and computer science educators when they have to choose an assessment system to be integrated in their e-Learning environment.


  1. Ala-Mutka, K., 2005. A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2):83-102.
  2. Blumenstein M., Green S., Nguyen A. and Muthukkumarasamy V., 2004. An experimental analysis of GAME: a generic automated marking environment. In Proceedings of the 9th annual SIGCSE conference on Innovation and technology in computer science education, Leeds, United Kingdom, pp 67-71
  3. Cheang B., Kurnia A., Lim A. and Oon W. C., 2003. On automated grading of programming assignments in an academic institution. In Computer Education, vol. 41, pp. 121-131.
  4. Douce C., Livingstone D., Orwell J., 2005. Automatic test-based assessment of programming: a review. In JERIC - Journal of Educational Resources in Computing, 5(3):4.
  5. Edwards S. H. and Pugh W., 2006. Toward a common automated grading platform. In SIGCSE 7806: Proceedings of the 37th SIGCSE technical symposium on Computer science education, (New York, NY, USA), ACM.
  6. Engels S., Lakshmanan V. and Craig M., 2007. Plagiarism detection using featurebased neural networks. In SIGCSE, pp. 34-38, 2007
  7. Higgins, C. A., Gray, G., Symeonidis, P., Tsintsifas, A., 2005. Automated assessment and experiences of teaching programming. In Journal on Educational Resources in Computing (JERIC), 5(3).
  8. Ihantola, P., Ahoniemi, T., Karavirta, V. And Seppälä, O., 2010. Review of recent systems for automatic assessment of programming assignments. In Koli Calling 7810 Proceedings of the 10th Koli Calling International Conference on Computing Education Research. ACM.
  9. Jackson, D. and Usher, M., 1997. Grading student programming using ASSYST. In Proceedings of 28th ACM SIGCSE Tech. Symposium on Computer Science Education, San Jose, California, USA, pp 335-339.
  10. Jena, S., 2008. Authoring and Sharing of Programming Exercises. In Master's Projects. Paper 19.
  11. Juedes, D. W., 2003. Experiences in Web-Based Grading. In 33rd ASEE/IEEE Frontiers in Education Conference November 5-8, 2003, Boulder, CO
  12. Leal, J. P. and Silva F., 2003. Mooshak: a Web-based multi-site programming contest system. In Software Practice & Experience, Volume 33 , Issue 6. Pages: 567 - 581, 2003, ISSN:0038-0644
  13. Leal, J. P., Queirós, R. and Ferreira D., 2010. Specifying a programming exercises evaluation service on the eFramework. In Xiangfeng Luo, Marc Spaniol, Lizhe Wang, Qing Li, Wolfgang Nejdl and Wu Zhang (Eds), Advances in Web-Based Learning - ICWL 2010 - 9th Internation Conference, Shanghai, China, December, 2010, LNCS 6483, pp. 141-150, ISBN 978-3-642- 17406-3
  14. Liang, Y., Liu, Q., Xu, J. and Wang, D., 2009. The recent development of automated programming assessment. In CISE - Computational Intelligence and Software Engineering, 1-5.
  15. Luck M. and Joy M., 1999. A secure on-line submission system. In Software - Practice and Experience, 29(8), pp721--740
  16. Malmi, L., Karavirta, V., Korhonen, A., Nikander, J., 2005. Experiences on automatically assessed algorithm simulation exercises with different resubmission policies. In Journal on Educational Resources in Computing (JERIC), 5(3).
  17. Mandal C., Sinha, V. L. and Reade C. M. P., 2004. A Web-Based Course Management Tool and Web Services. In Electronic Journal of E-Learning, Vol 2(1) paper no. 19
  18. Mandal A. K., Mandal C. and Reade C. M. P., 2006. Architecture Of An Automatic Program Evaluation System. In CSIE Proceedings
  19. Mansouri, F., Cleveland, A. Gibbon, Colin, A. Higgins. PRAM: prolog automatic marker. In Proceedings of ITiCSE'1998. pp.166170
  20. Pisan Y., Richards D., Sloane A., Koncek H. and Mitchell S., 2003. Submit! A Web-Based System for Automatic Program Critiquing. In Proceedings of the Fifth Australasian Computing Education Conference (ACE 2003), Adelaide, Australia, Australian Computer Society, pp. 59-68.
  21. Queirós R. and Leal J. P., 2011. A Survey on eLearning Content Standardization. In 4th World Summit on the Knowledge Society, Mykonos, Greece.
  22. Reek, K. A., 1989.The TRY system or how to avoid testing student programs. In Proceedings of SIGCSE, pp 112-116.
  23. Rehak, D. R., Mason, R., 2003. Keeping the learning in learning objects. In Littlejohn, A. (Ed.) Reusing online resources: a sustainable approach to e-Learning. Kogan Page, London, 2003. (pp.22-30).
  24. Romli, R., Sulaiman, S. and Zamli, K. Z., 2010. Automatic programming assessment and test data generation a review on its approaches. In Information Technology (ITSim), 2010 International Symposium in. DOI: 10.1109/ITSIM.2010.5561488
  25. Saikkonen R., Malmi L., Korhonen A., 2001. Fully automatic assessment of programming exercises. In Proceedings of the 6th annual conference on Innovation and Technology in Computer Science Education (ITiCSE), pp. 133-136, 2001
  26. Striewe M. and Goedicke M., 2010. Visualizing Data Structures in an E-Learning System. In Proceedings of the 2nd International Conference on Computer Supported Education (CSEDU) 2010, Valencia, Spain, volume 1, pages 172-179, 2010
  27. Tang, C. M., Yu, Y. T., and Poon, C. K., 2009a. Automated systems for testing student programs: Practical issues and requirements. In Proceedings of the International Workshop on Strategies for Practical Integration of Emerging and Contemporary Technologies in Assessment and Learning, pp. 132±136
  28. Tang, C. M., Yu, Y. T., and Poon, C. K., 2009b. An approach towards automatic testing of student programs using token patterns. In Proceedings of the 17th International Conference on Computers in Education (ICCE 2009), pp. 188±190
  29. Tang C. M., Yu Y. T., Poon C. K., 2010. A Review of the Strategies for Output Correctness Determination in Automated Assessment of Student Programs. In Proceedings of Global Chinese Conference on Computers in Education.
  30. Traetteberg H., Aalberg T., 2006. JExercise: A specification-based and test-driven exercise support plugin for Eclipse. In Proceedings of the 2006 OOPSLA Workshop on Eclipse Technology eXchange, ETX 2006 (2006), 70-74.
  31. Verdú E., Regueras L. M., Verdú M. J., Leal J. P., Castro J. P. and Queirós R., 2011. A Distributed System for Learning Programming On-line. In Computers & Education Jornal, 2011, ISSN 0360-1315

Paper Citation

in Harvard Style

Queirós R. and Leal J. (2012). PROGRAMMING EXERCISES EVALUATION SYSTEMS - An Interoperability Survey . In Proceedings of the 4th International Conference on Computer Supported Education - Volume 1: CSEDU, ISBN 978-989-8565-06-8, pages 83-90. DOI: 10.5220/0003924900830090

in Bibtex Style

author={Ricardo Queirós and José Paulo Leal},
booktitle={Proceedings of the 4th International Conference on Computer Supported Education - Volume 1: CSEDU,},

in EndNote Style

JO - Proceedings of the 4th International Conference on Computer Supported Education - Volume 1: CSEDU,
SN - 978-989-8565-06-8
AU - Queirós R.
AU - Leal J.
PY - 2012
SP - 83
EP - 90
DO - 10.5220/0003924900830090