Toward Preventing Stack Overflow Using Kernel Properties

Benjamin Teissier, Stefan D. Bruda


We contribute to the investigation of buffer overflows by finding a more accurate way of preventing their exploitation. We work at the highest privilege levels and in the safest part of a GNU/Linux system, namely the kernel. We provide a system that allows the kernel to detect overflows and prevent their exploitation. The kernel injects at launch time some (minimal) code into the binary being run, and subsequently uses this code to monitor the execution of that program with respect to its stack use, thus detecting stack overflows. The system stands alone in the sense that it does not need any hardware support; it also works on any program, no matter how that program was conceived or compiled. Beside the theoretical concepts we also present a proof-of-concept patch to the kernel supporting our idea. Overall we effectively show that guarding against buffer overflows at run time is not only possible but also feasible. In addition we take the first steps toward implementing such a defense.


  1. Anderson, J. P. (1972). planning study. ande72.pdf, p. 61.
  2. ASLR (2003). Address space layout randomization. (retrieved Nov. 2012).
  3. Bulba and Kil3r (2000). Bypassing stackguard and stackshield. Phrack, 10(56). =56&id=5.
  4. c0ntex (2012). Bypassing non-executable-stack during exploitation using return-to-libc. www.infosecwriters. com/text resources/pdf/return-to-libc.pdf (retrieved Nov. 2012).
  5. CERT/CC (2002). Advisory CA-2001-19 “Code Red” worm exploiting buffer overflow in IIS indexing service DLL. (retrieved Sep. 2013).
  6. Delikon (2004). Changing the per-file entry-point to avoid anti-virus detection.
  7. Etoh, H. (2001). gcc-2.95.3).
  8. (retrieved Nov. 2012).
  9. Goodin, D. (2013). Puzzle box: The quest to crack the world's most mysterious malware warhead. Arstechnica.
  10. Intel (2013). Intel 64 and IA-32 architectures software developer's manual combined volumes 2A, 2B, and 2C: Instruction set reference, A-Z.
  11. Kerouanton, B. (2012). Reinventing old school vulnerabilities. (retrieved Nov. 2012).
  12. Knowles, D. (2007). W32.SQLExp.Worm. com/security response/writeup.jsp?docid=2003- 012502-3306-99 (retrieved Sep. 2013).
  13. Levy, E. (1996). Smashing the stack for fun and profit. Phrack, 7(49). =49&id=14.
  14. Noexec (2003). Non-executable pages design and implementation. (retrieved Nov. 2012).
  15. Padmanabhuni, B. M. and Tan, H. B. K. (2011). Defending against buffer overflow vulnerabilities. Computer, 44(11):53-60.
  16. Panchamukhi, P. (2004). Kernel debugging with kprobes. IBM DevelopersWorks. developerworks/library/l-kprobes/index.html.
  17. Rascagneres, P. (2012). Voyage au centre du SSP-Linux. au centre du ssp linux (retrieved Nov. 2012).
  18. Seeley, D. (2007). A tour of the worm. web/20070520233435/˜franl/ worm.html (retrieved Nov. 2012).
  19. Shacham, H., Page, M., B. Pfaff, E.-J., Goh, Modadugu, N., and Boneh, D. (2004). On the effectiveness of address-space randomization. In Proceedings of the 11th ACM Conference on Computer and Communications Security, pages 298-307.
  20. Shao, Z., Cao, J., Chan, K. C. C., Xue, C., and H.-M.Sha, E. (2006). Hardware/software optimization for array & pointer boundary checking against buffer overflow attacks. Journal of Parallel and Distributed Computing, 66(9):1129-1136.
  21. Shao, Z., Xue, C., Zhuge, Q., and Sha, E. H.-M. (2004). Security protection and checking in embedded system integration against buffer overflow attacks. In Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC 2004), volume I, pages 409-413.
  22. Teissier, B. and Bruda, S. D. (2014). An approach to stack overflow counter-measures using kernel properties. Technical Report 2014-001, Department of Computer Science, Bishop's University. media/papers/bucstr-2014-001.pdf.

Paper Citation

in Harvard Style

Teissier B. and Bruda S. (2014). Toward Preventing Stack Overflow Using Kernel Properties . In Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014) ISBN 978-989-758-036-9, pages 369-376. DOI: 10.5220/0005097803690376

in Bibtex Style

author={Benjamin Teissier and Stefan D. Bruda},
title={Toward Preventing Stack Overflow Using Kernel Properties},
booktitle={Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014)},

in EndNote Style

JO - Proceedings of the 9th International Conference on Software Engineering and Applications - Volume 1: ICSOFT-EA, (ICSOFT 2014)
TI - Toward Preventing Stack Overflow Using Kernel Properties
SN - 978-989-758-036-9
AU - Teissier B.
AU - Bruda S.
PY - 2014
SP - 369
EP - 376
DO - 10.5220/0005097803690376