Investigating the Difficulty of Commercial-level Compiler Warning Messages for Novice Programmers

Yoshitaka Kojima, Yoshitaka Arahori, Katsuhiko Gondow

2015

Abstract

Many researchers refer to the folklore “warning messages in commercial-level compilers like GCC are difficult for novice programmers, which leads to low learning efficiency.” However, there is little quantitative investigation about this, so it is still unknown if (and to what extent) the warning messages are really difficult. In this paper, we provide a quantitative investigation about the difficulty of the warning messages. More specifically, as a sample code set we first collected 90 small programs in C language that are error-prone for novice programmers. Then we performed the investigation on the warning emission and its difficulty for 4 compilers and 5 static analysis tools, which are all commercial-level, using the sample code set. The difficulty of warning messages were evaluated by 7 students as research participants, using 4 evaluation criteria of clarity, specificity, constructive guidance, and plain terminology. As a result, we obtained several important quantitative findings: e.g., the deviation of warning emission presence in compilers and static analysis tools is large; and 35.7% of warning messages lack clarity, and 35.9% of warning messages lack specificity, which suggests roughly one third of warning messages are difficult for novice programmers to understand.

References

  1. Dy, T. and Rodrigo, M. M. (2010). A detector for non-literal java errors. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli Calling 7810, pages 118-122, New York, NY, USA. ACM.
  2. Flowers, T., Carver, C., and Jackson, J. (2004). Empowering students and building confidence in novice programmers through gauntlet. In Frontiers in Education, 2004. FIE 2004. 34th Annual, pages T3H/10-T3H/13 Vol. 1.
  3. Gondow, K. (2015). Sample code set, all compiler warning messages and anonymized result of questionnaire. http://www.sde.cs.titech.ac.jp/cm/.
  4. Gross, P. and Powers, K. (2005). Evaluating assessments of novice programming environments. In Proceedings of the First International Workshop on Computing Education Research, ICER 7805, pages 99-110, New York, NY, USA. ACM.
  5. Horning, J. J. (1976). What the compiler should tell the user. In Compiler Construction, An Advanced Course, 2Nd Ed., pages 525-548, London, UK, UK. SpringerVerlag.
  6. Hristova, M., Misra, A., Rutter, M., and Mercuri, R. (2003). Identifying and correcting java programming errors for introductory computer science students. SIGCSE Bull., 35(1):153-156.
  7. Jackson, J., Cobb, M., and Carver, C. (2005). Identifying top java errors for novice programmers. In Frontiers in Education, 2005. FIE 7805. Proceedings 35th Annual Conference, pages T4C-T4C.
  8. Kölling, M., Quig, B., Patterson, A., and Rosenberg, J. (2003). The bluej system and its pedagogy. Computer Science Education, 13(4):249-268.
  9. Kummerfeld, S. K. and Kay, J. (2003). The neglected battle fields of syntax errors. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20, ACE 7803, pages 105-111, Darlinghurst, Australia, Australia. Australian Computer Society, Inc.
  10. Marceau, G., Fisler, K., and Krishnamurthi, S. (2011a). Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE 7811, pages 499-504, New York, NY, USA. ACM.
  11. Marceau, G., Fisler, K., and Krishnamurthi, S. (2011b). Mind your language: On novices' interactions with error messages. In Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2011, pages 3-18, New York, NY, USA. ACM.
  12. Nienaltowski, M.-H., Pedroni, M., and Meyer, B. (2008). Compiler error messages: What can help novices? In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, SIGCSE 7808, pages 168-172, New York, NY, USA. ACM.
  13. Pears, A., Seidman, S., Malmi, L., Mannila, L., Adams, E., Bennedsen, J., Devlin, M., and Paterson, J. (2007). A survey of literature on the teaching of introductory programming. SIGCSE Bull., 39(4):204-223.
  14. Traver, V. J. (2010). On compiler error messages: What they say and what they mean. Adv. in Hum.-Comp. Int., 2010:3:1-3:26.
Download


Paper Citation


in Harvard Style

Kojima Y., Arahori Y. and Gondow K. (2015). Investigating the Difficulty of Commercial-level Compiler Warning Messages for Novice Programmers . In Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-108-3, pages 483-490. DOI: 10.5220/0005437404830490


in Bibtex Style

@conference{csedu15,
author={Yoshitaka Kojima and Yoshitaka Arahori and Katsuhiko Gondow},
title={Investigating the Difficulty of Commercial-level Compiler Warning Messages for Novice Programmers},
booktitle={Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2015},
pages={483-490},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005437404830490},
isbn={978-989-758-108-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 7th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Investigating the Difficulty of Commercial-level Compiler Warning Messages for Novice Programmers
SN - 978-989-758-108-3
AU - Kojima Y.
AU - Arahori Y.
AU - Gondow K.
PY - 2015
SP - 483
EP - 490
DO - 10.5220/0005437404830490